2022-11-23T01:39:17.9506461Z Requested labels: linux.16xlarge.nvidia.gpu
2022-11-23T01:39:17.9506532Z Job defined at: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/master
2022-11-23T01:39:17.9506555Z Waiting for a runner to pick up this job...
2022-11-23T01:39:18.1369971Z Job is about to start running on the runner: i-018c617fc63e71812 (organization)
2022-11-23T01:39:22.3471673Z Current runner version: '2.299.1'
2022-11-23T01:39:22.3479125Z Runner name: 'i-018c617fc63e71812'
2022-11-23T01:39:22.3479788Z Runner group name: 'Default'
2022-11-23T01:39:22.3480613Z Machine name: 'ip-10-0-2-152'
2022-11-23T01:39:22.3483445Z ##[group]GITHUB_TOKEN Permissions
2022-11-23T01:39:22.3484259Z Actions: write
2022-11-23T01:39:22.3484802Z Checks: write
2022-11-23T01:39:22.3485182Z Contents: write
2022-11-23T01:39:22.3485576Z Deployments: write
2022-11-23T01:39:22.3485963Z Discussions: write
2022-11-23T01:39:22.3486336Z Issues: write
2022-11-23T01:39:22.3486670Z Metadata: read
2022-11-23T01:39:22.3487083Z Packages: write
2022-11-23T01:39:22.3487462Z Pages: write
2022-11-23T01:39:22.3487794Z PullRequests: write
2022-11-23T01:39:22.3488223Z RepositoryProjects: write
2022-11-23T01:39:22.3488681Z SecurityEvents: write
2022-11-23T01:39:22.3489020Z Statuses: write
2022-11-23T01:39:22.3489389Z ##[endgroup]
2022-11-23T01:39:22.3493311Z Secret source: Actions
2022-11-23T01:39:22.3494033Z Prepare workflow directory
2022-11-23T01:39:22.4777724Z Prepare all required actions
2022-11-23T01:39:22.4995474Z Getting action download info
2022-11-23T01:39:22.7346692Z Download action repository 'pytorch/test-infra@main' (SHA:c57ff4d9a93667a5571a80a0e92c3e2674aeedfd)
2022-11-23T01:39:23.0585101Z Download action repository 'pytorch/pytorch@master' (SHA:1cfd3858ac54fe3883534309081631a0a892ba3f)
2022-11-23T01:39:26.3603111Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2022-11-23T01:39:26.7433941Z Getting action download info
2022-11-23T01:39:26.8975276Z Download action repository 'malfet/checkout@silent-checkout' (SHA:c7b8fef48edfe1bca0044a44b1f7f7c4318a3076)
2022-11-23T01:39:27.1374467Z Getting action download info
2022-11-23T01:39:27.2884922Z Download action repository 'nick-fields/retry@7d4a37704547a311dbb66ebdf5b23ec19374a767' (SHA:7d4a37704547a311dbb66ebdf5b23ec19374a767)
2022-11-23T01:39:27.4333746Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml
2022-11-23T01:39:27.4336469Z ##[group] Inputs
2022-11-23T01:39:27.4337383Z   build-environment: linux-bionic-cuda11.6-py3.9-gcc7
2022-11-23T01:39:27.4337890Z   test-matrix: { include: [
  { config: "multigpu", shard: 1, num_shards: 1, runner: "linux.16xlarge.nvidia.gpu" },
]}

2022-11-23T01:39:27.4338544Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:39:27.4339035Z   sync-tag: 
2022-11-23T01:39:27.4340217Z   timeout-minutes: 240
2022-11-23T01:39:27.4340679Z ##[endgroup]
2022-11-23T01:39:27.4341559Z Complete job name: linux-bionic-cuda11.6-py3.9-gcc7 / test (multigpu, 1, 1, linux.16xlarge.nvidia.gpu, rerun_disabled_tests)
2022-11-23T01:39:27.5694089Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main
2022-11-23T01:39:27.5694491Z with:
2022-11-23T01:39:27.5695075Z   github-secret: ***
2022-11-23T01:39:27.5695374Z   activate-with-label: false
2022-11-23T01:39:27.5695771Z   label: with-ssh
2022-11-23T01:39:27.5696045Z   remove-existing-keys: true
2022-11-23T01:39:27.5696317Z env:
2022-11-23T01:39:27.5696571Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:39:27.5697108Z ##[endgroup]
2022-11-23T01:39:27.6787170Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys
2022-11-23T01:39:27.7021021Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@master
2022-11-23T01:39:27.7021405Z with:
2022-11-23T01:39:27.7021625Z   submodules: recursive
2022-11-23T01:39:27.7021892Z   fetch-depth: 0
2022-11-23T01:39:27.7022167Z env:
2022-11-23T01:39:27.7022381Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:39:27.7022748Z ##[endgroup]
2022-11-23T01:39:27.7315260Z ##[group]Run retry () {
2022-11-23T01:39:27.7315596Z retry () {
2022-11-23T01:39:27.7315893Z   $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
2022-11-23T01:39:27.7316194Z }
2022-11-23T01:39:27.7316615Z echo "${GITHUB_WORKSPACE}"
2022-11-23T01:39:27.7317082Z if [ -z "${NO_SUDO}" ]; then
2022-11-23T01:39:27.7317578Z   retry sudo rm -rf "${GITHUB_WORKSPACE}"
2022-11-23T01:39:27.7317870Z else
2022-11-23T01:39:27.7318132Z   retry rm -rf "${GITHUB_WORKSPACE}"
2022-11-23T01:39:27.7318408Z fi
2022-11-23T01:39:27.7318763Z mkdir "${GITHUB_WORKSPACE}"
2022-11-23T01:39:27.7338230Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:39:27.7338540Z env:
2022-11-23T01:39:27.7338795Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:39:27.7339051Z   NO_SUDO: 
2022-11-23T01:39:27.7339267Z ##[endgroup]
2022-11-23T01:39:27.7467435Z /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-11-23T01:39:27.7842036Z ##[group]Run malfet/checkout@silent-checkout
2022-11-23T01:39:27.7842340Z with:
2022-11-23T01:39:27.7842785Z   ref: 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:39:27.7843050Z   fetch-depth: 0
2022-11-23T01:39:27.7843309Z   submodules: recursive
2022-11-23T01:39:27.7843583Z   quiet-checkout: true
2022-11-23T01:39:27.7843842Z   repository: pytorch/pytorch
2022-11-23T01:39:27.7844284Z   token: ***
2022-11-23T01:39:27.7844716Z   ssh-strict: true
2022-11-23T01:39:27.7845000Z   persist-credentials: true
2022-11-23T01:39:27.7845253Z   clean: true
2022-11-23T01:39:27.7845659Z   lfs: false
2022-11-23T01:39:27.7845911Z   set-safe-directory: true
2022-11-23T01:39:27.7846136Z env:
2022-11-23T01:39:27.7846372Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:39:27.7846626Z ##[endgroup]
2022-11-23T01:39:27.9368055Z Syncing repository: pytorch/pytorch
2022-11-23T01:39:27.9369936Z ##[group]Getting Git version info
2022-11-23T01:39:27.9370492Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-11-23T01:39:27.9371098Z [command]/usr/bin/git version
2022-11-23T01:39:27.9371405Z git version 2.37.1
2022-11-23T01:39:27.9392206Z ##[endgroup]
2022-11-23T01:39:27.9414892Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/3f5e92ce-8de0-4462-9154-366f534ee703' before making global git config changes
2022-11-23T01:39:27.9415513Z Adding repository directory to the temporary git global config as a safe directory
2022-11-23T01:39:27.9423281Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-11-23T01:39:27.9470172Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-11-23T01:39:27.9477333Z ##[group]Initializing the repository
2022-11-23T01:39:27.9479829Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-11-23T01:39:27.9514181Z hint: Using 'master' as the name for the initial branch. This default branch name
2022-11-23T01:39:27.9514749Z hint: is subject to change. To configure the initial branch name to use in all
2022-11-23T01:39:27.9515388Z hint: of your new repositories, which will suppress this warning, call:
2022-11-23T01:39:27.9515733Z hint: 
2022-11-23T01:39:27.9516129Z hint: 	git config --global init.defaultBranch <name>
2022-11-23T01:39:27.9516467Z hint: 
2022-11-23T01:39:27.9516847Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2022-11-23T01:39:27.9517274Z hint: 'development'. The just-created branch can be renamed via this command:
2022-11-23T01:39:27.9517712Z hint: 
2022-11-23T01:39:27.9518080Z hint: 	git branch -m <name>
2022-11-23T01:39:27.9518594Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/
2022-11-23T01:39:27.9528544Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2022-11-23T01:39:27.9571655Z ##[endgroup]
2022-11-23T01:39:27.9572219Z ##[group]Disabling automatic garbage collection
2022-11-23T01:39:27.9577073Z [command]/usr/bin/git config --local gc.auto 0
2022-11-23T01:39:27.9610883Z ##[endgroup]
2022-11-23T01:39:27.9611770Z ##[group]Setting up auth
2022-11-23T01:39:27.9620556Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-11-23T01:39:27.9657675Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-11-23T01:39:28.0027856Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-11-23T01:39:28.0062078Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-11-23T01:39:28.0413711Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-11-23T01:39:28.0466232Z ##[endgroup]
2022-11-23T01:39:28.0466725Z ##[group]Fetching the repository
2022-11-23T01:39:28.0475617Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --quiet --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2022-11-23T01:40:20.0004296Z [command]/usr/bin/git rev-parse --verify --quiet 1cfd3858ac54fe3883534309081631a0a892ba3f^{object}
2022-11-23T01:40:20.0034456Z 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:40:20.0040306Z ##[endgroup]
2022-11-23T01:40:20.0042728Z ##[group]Determining the checkout info
2022-11-23T01:40:20.0043209Z ##[endgroup]
2022-11-23T01:40:20.0043674Z ##[group]Checking out the ref
2022-11-23T01:40:20.0045987Z [command]/usr/bin/git checkout --quiet --force 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:40:21.8175836Z ##[endgroup]
2022-11-23T01:40:21.8176521Z ##[group]Setting up auth for fetching submodules
2022-11-23T01:40:21.8182281Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-11-23T01:40:21.8244455Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2022-11-23T01:40:21.8277857Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2022-11-23T01:40:21.8319981Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2022-11-23T01:40:21.8356471Z ##[endgroup]
2022-11-23T01:40:21.8356953Z ##[group]Fetching submodules
2022-11-23T01:40:21.8362291Z [command]/usr/bin/git submodule sync --recursive
2022-11-23T01:40:21.8738023Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2022-11-23T01:40:21.9087711Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2022-11-23T01:40:21.9088813Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2022-11-23T01:40:21.9092287Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2022-11-23T01:40:21.9094720Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2022-11-23T01:40:21.9099862Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK'
2022-11-23T01:40:21.9102244Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator'
2022-11-23T01:40:21.9107657Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2022-11-23T01:40:21.9109531Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2022-11-23T01:40:21.9114831Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2022-11-23T01:40:21.9117009Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub'
2022-11-23T01:40:21.9122929Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2022-11-23T01:40:21.9125086Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass'
2022-11-23T01:40:21.9131144Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen'
2022-11-23T01:40:21.9133813Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2022-11-23T01:40:21.9140234Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2022-11-23T01:40:21.9144254Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2022-11-23T01:40:21.9149579Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi'
2022-11-23T01:40:21.9154164Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:40:21.9159482Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo'
2022-11-23T01:40:21.9164881Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2022-11-23T01:40:21.9170243Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2022-11-23T01:40:21.9175903Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake'
2022-11-23T01:40:21.9181721Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi'
2022-11-23T01:40:21.9187120Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2022-11-23T01:40:21.9192850Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl'
2022-11-23T01:40:21.9199095Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse'
2022-11-23T01:40:21.9205539Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann'
2022-11-23T01:40:21.9212075Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2022-11-23T01:40:21.9218633Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt'
2022-11-23T01:40:21.9225572Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2022-11-23T01:40:21.9232345Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2022-11-23T01:40:21.9239249Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2022-11-23T01:40:21.9246328Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2022-11-23T01:40:21.9253409Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2022-11-23T01:40:21.9260679Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum'
2022-11-23T01:40:21.9268294Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy'
2022-11-23T01:40:21.9276336Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six'
2022-11-23T01:40:21.9283766Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2022-11-23T01:40:21.9291751Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb'
2022-11-23T01:40:21.9300714Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2022-11-23T01:40:21.9308700Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd'
2022-11-23T01:40:21.9342877Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2022-11-23T01:40:22.1949670Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2022-11-23T01:40:22.4474306Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2022-11-23T01:40:22.7471490Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2022-11-23T01:40:23.0591593Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'...
2022-11-23T01:40:23.3411828Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'...
2022-11-23T01:40:25.2725170Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2022-11-23T01:40:30.9644284Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2022-11-23T01:40:31.3780733Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2022-11-23T01:40:31.9377888Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'...
2022-11-23T01:40:33.4947992Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2022-11-23T01:40:34.8436399Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'...
2022-11-23T01:40:36.5280727Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'...
2022-11-23T01:40:43.8859408Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2022-11-23T01:40:44.6360051Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2022-11-23T01:40:46.1395145Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2022-11-23T01:40:47.2100416Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'...
2022-11-23T01:40:47.4178168Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2022-11-23T01:40:47.9022467Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2022-11-23T01:40:48.2240061Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2022-11-23T01:40:49.2134798Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2022-11-23T01:40:49.7034808Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'...
2022-11-23T01:40:49.9042803Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'...
2022-11-23T01:40:50.1704888Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2022-11-23T01:40:51.7701343Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'...
2022-11-23T01:40:52.2343305Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'...
2022-11-23T01:40:52.6303677Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'...
2022-11-23T01:40:58.7486845Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2022-11-23T01:41:00.3191592Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'...
2022-11-23T01:41:00.7770547Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2022-11-23T01:41:01.0776347Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2022-11-23T01:41:07.2338977Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2022-11-23T01:41:07.4437808Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2022-11-23T01:41:07.7316264Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2022-11-23T01:41:08.6137961Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-enum'...
2022-11-23T01:41:08.8529774Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2022-11-23T01:41:09.1839401Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-six'...
2022-11-23T01:41:09.4981263Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2022-11-23T01:41:10.0908343Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'...
2022-11-23T01:41:12.8846775Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2022-11-23T01:41:13.4128077Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'...
2022-11-23T01:41:16.6687380Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2022-11-23T01:41:16.6830938Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2022-11-23T01:41:16.6940184Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2022-11-23T01:41:16.7242037Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2022-11-23T01:41:16.7533885Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c'
2022-11-23T01:41:16.8010659Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191'
2022-11-23T01:41:17.5750076Z Submodule path 'third_party/XNNPACK': checked out 'ae108ef49aa5623b896fc93d4298c49d1750d9ba'
2022-11-23T01:41:17.6007389Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2022-11-23T01:41:17.7268896Z Submodule path 'third_party/cpuinfo': checked out '8ec7bd91ad0470e61cf38f618cc1f270dede599c'
2022-11-23T01:41:17.7711834Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4'
2022-11-23T01:41:18.1406959Z Submodule path 'third_party/cudnn_frontend': checked out '171a7a986f7fbd9ed71bd0cf3c7ad4f55843d6b3'
2022-11-23T01:41:18.6767072Z Submodule path 'third_party/cutlass': checked out 'b72cbf957df8cf84a6d0ff91c190ad51a9c1d24a'
2022-11-23T01:41:18.9850796Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2022-11-23T01:41:19.0435948Z Submodule path 'third_party/fbgemm': checked out '4d1738b3142a6cb0c032cd639e239566010b054a'
2022-11-23T01:41:19.0460208Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:41:19.0462552Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:41:19.0466513Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:41:19.0470777Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:41:19.0503376Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'...
2022-11-23T01:41:20.0168632Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'...
2022-11-23T01:41:20.5867803Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'...
2022-11-23T01:41:21.5828209Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'...
2022-11-23T01:41:21.9513042Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81'
2022-11-23T01:41:22.0804393Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2022-11-23T01:41:22.1558614Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2022-11-23T01:41:22.1685754Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '1840658c184f3eeba787dae0f06c45756c1daaf5'
2022-11-23T01:41:22.2909102Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa'
2022-11-23T01:41:22.3359002Z Submodule path 'third_party/fmt': checked out '7bdf0628b1276379886c7f6dda2cef2b3b374f0b'
2022-11-23T01:41:22.3475983Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad'
2022-11-23T01:41:22.3975595Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2022-11-23T01:41:22.4279863Z Submodule path 'third_party/gloo': checked out '4a5e339b764261d20fc409071dc7a8b8989aa195'
2022-11-23T01:41:22.4850302Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2022-11-23T01:41:22.4996788Z Submodule path 'third_party/ideep': checked out '5ddc65efe0428bbce2942b3ce5e3ce15239abe2f'
2022-11-23T01:41:22.5017319Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2022-11-23T01:41:22.5047640Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2022-11-23T01:41:31.0422339Z Submodule path 'third_party/ideep/mkl-dnn': checked out 'd19d0f795c60695bd32f894c6f01771b2dfbe24d'
2022-11-23T01:41:31.0442823Z Submodule 'third_party/oneDNN' (https://github.com/oneapi-src/oneDNN.git) registered for path 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:41:31.0476155Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN'...
2022-11-23T01:41:39.6354384Z Submodule path 'third_party/ideep/mkl-dnn/third_party/oneDNN': checked out '650085b2f3643aad05c629425983491d63b5c289'
2022-11-23T01:41:39.6488678Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724'
2022-11-23T01:41:39.6675948Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42'
2022-11-23T01:41:39.7838478Z Submodule path 'third_party/kineto': checked out '6c1629809068efd78a8d56b4aa479c7ec49ae562'
2022-11-23T01:41:39.7859447Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:41:39.7861152Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:41:39.7894677Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2022-11-23T01:41:40.8930785Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2022-11-23T01:41:41.9364977Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '2591ab91c3898c9f6544fff04660276537d32ffd'
2022-11-23T01:41:42.0051978Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2022-11-23T01:41:42.0320033Z Submodule path 'third_party/nccl/nccl': checked out 'f89fd4777d2ef9229c039ff750ae21da01626f52'
2022-11-23T01:41:42.0490036Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a'
2022-11-23T01:41:42.1876404Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806'
2022-11-23T01:41:42.5131258Z Submodule path 'third_party/onnx': checked out 'f7ee1ac60d06abe8e26c9b6bbe1e3db5286b614b'
2022-11-23T01:41:42.5165504Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark'
2022-11-23T01:41:42.5166983Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2022-11-23T01:41:42.5198032Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'...
2022-11-23T01:41:42.9191534Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2022-11-23T01:41:43.7876948Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2022-11-23T01:41:43.8289284Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'ffa346860b306c9bbfb341aed9c14c067751feb8'
2022-11-23T01:41:43.8487447Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f'
2022-11-23T01:41:43.8506390Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:41:43.8535621Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'...
2022-11-23T01:41:45.6069067Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8'
2022-11-23T01:41:45.6098062Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:41:45.6100532Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:41:45.6136229Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'...
2022-11-23T01:41:46.0267764Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'...
2022-11-23T01:41:46.8771878Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508'
2022-11-23T01:41:46.9596744Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c'
2022-11-23T01:41:46.9615825Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:41:46.9646936Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'...
2022-11-23T01:41:47.1828652Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-11-23T01:41:47.1939173Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a'
2022-11-23T01:41:47.5202664Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2022-11-23T01:41:47.5228781Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:41:47.5230338Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2022-11-23T01:41:47.5266493Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2022-11-23T01:41:47.9336133Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2022-11-23T01:41:48.9145520Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2022-11-23T01:41:48.9993151Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2022-11-23T01:41:49.0107092Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2022-11-23T01:41:49.0248469Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413'
2022-11-23T01:41:49.0679794Z Submodule path 'third_party/pybind11': checked out '80dc998efced8ceb2be59756668a7e90e8bef917'
2022-11-23T01:41:49.0793733Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7'
2022-11-23T01:41:49.1149333Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2022-11-23T01:41:49.1268016Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a'
2022-11-23T01:41:49.1822382Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff'
2022-11-23T01:41:49.3248021Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9'
2022-11-23T01:41:49.3574459Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2022-11-23T01:41:49.3594008Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:41:49.3595406Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:41:49.3598980Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:41:49.3601684Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:41:49.3636484Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2022-11-23T01:41:50.3595790Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2022-11-23T01:41:50.6319480Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2022-11-23T01:41:53.3340535Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2022-11-23T01:41:54.2922182Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2022-11-23T01:41:54.3099735Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2022-11-23T01:41:54.3926662Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2022-11-23T01:41:54.4269428Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2022-11-23T01:41:54.4289356Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:41:54.4320300Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2022-11-23T01:41:54.6658090Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-11-23T01:41:54.8303794Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8'
2022-11-23T01:41:54.8353949Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2022-11-23T01:41:54.8740064Z Entering 'android/libs/fbjni'
2022-11-23T01:41:54.8789979Z Entering 'third_party/FP16'
2022-11-23T01:41:54.8842403Z Entering 'third_party/FXdiv'
2022-11-23T01:41:54.8889322Z Entering 'third_party/NNPACK'
2022-11-23T01:41:54.8940505Z Entering 'third_party/QNNPACK'
2022-11-23T01:41:54.8988152Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T01:41:54.9036606Z Entering 'third_party/XNNPACK'
2022-11-23T01:41:54.9095751Z Entering 'third_party/benchmark'
2022-11-23T01:41:54.9148354Z Entering 'third_party/cpuinfo'
2022-11-23T01:41:54.9206507Z Entering 'third_party/cub'
2022-11-23T01:41:54.9257605Z Entering 'third_party/cudnn_frontend'
2022-11-23T01:41:54.9319536Z Entering 'third_party/cutlass'
2022-11-23T01:41:54.9379807Z Entering 'third_party/eigen'
2022-11-23T01:41:54.9432471Z Entering 'third_party/fbgemm'
2022-11-23T01:41:54.9482787Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:41:54.9529156Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:41:54.9581702Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:41:54.9632704Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:41:54.9683977Z Entering 'third_party/flatbuffers'
2022-11-23T01:41:54.9734132Z Entering 'third_party/fmt'
2022-11-23T01:41:54.9788258Z Entering 'third_party/foxi'
2022-11-23T01:41:54.9839716Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:41:54.9890998Z Entering 'third_party/gloo'
2022-11-23T01:41:54.9945264Z Entering 'third_party/googletest'
2022-11-23T01:41:54.9993581Z Entering 'third_party/ideep'
2022-11-23T01:41:55.0046492Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T01:41:55.0099389Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:41:55.0156875Z Entering 'third_party/ios-cmake'
2022-11-23T01:41:55.0208837Z Entering 'third_party/ittapi'
2022-11-23T01:41:55.0258598Z Entering 'third_party/kineto'
2022-11-23T01:41:55.0307465Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:41:55.0357404Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:41:55.0413848Z Entering 'third_party/nccl/nccl'
2022-11-23T01:41:55.0462869Z Entering 'third_party/neon2sse'
2022-11-23T01:41:55.0512731Z Entering 'third_party/nlohmann'
2022-11-23T01:41:55.0562708Z Entering 'third_party/onnx'
2022-11-23T01:41:55.0622994Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T01:41:55.0675749Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T01:41:55.0729481Z Entering 'third_party/onnx-tensorrt'
2022-11-23T01:41:55.0778901Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:41:55.0831952Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:41:55.0881594Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:41:55.0931445Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:41:55.0987502Z Entering 'third_party/pocketfft'
2022-11-23T01:41:55.1034125Z Entering 'third_party/protobuf'
2022-11-23T01:41:55.1088090Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:41:55.1138667Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T01:41:55.1192395Z Entering 'third_party/psimd'
2022-11-23T01:41:55.1243534Z Entering 'third_party/pthreadpool'
2022-11-23T01:41:55.1293174Z Entering 'third_party/pybind11'
2022-11-23T01:41:55.1345197Z Entering 'third_party/python-enum'
2022-11-23T01:41:55.1393349Z Entering 'third_party/python-peachpy'
2022-11-23T01:41:55.1443120Z Entering 'third_party/python-six'
2022-11-23T01:41:55.1492600Z Entering 'third_party/sleef'
2022-11-23T01:41:55.1543124Z Entering 'third_party/tbb'
2022-11-23T01:41:55.1593827Z Entering 'third_party/tensorpipe'
2022-11-23T01:41:55.1644271Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:41:55.1694271Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:41:55.1743313Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:41:55.1790169Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:41:55.1836355Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:41:55.1889738Z Entering 'third_party/zstd'
2022-11-23T01:41:55.1949754Z ##[endgroup]
2022-11-23T01:41:55.1952812Z ##[group]Persisting credentials for submodules
2022-11-23T01:41:55.1957197Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :
2022-11-23T01:41:55.2311221Z Entering 'android/libs/fbjni'
2022-11-23T01:41:55.2363306Z Entering 'third_party/FP16'
2022-11-23T01:41:55.2410082Z Entering 'third_party/FXdiv'
2022-11-23T01:41:55.2458141Z Entering 'third_party/NNPACK'
2022-11-23T01:41:55.2507186Z Entering 'third_party/QNNPACK'
2022-11-23T01:41:55.2555302Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T01:41:55.2606867Z Entering 'third_party/XNNPACK'
2022-11-23T01:41:55.2670205Z Entering 'third_party/benchmark'
2022-11-23T01:41:55.2718849Z Entering 'third_party/cpuinfo'
2022-11-23T01:41:55.2765943Z Entering 'third_party/cub'
2022-11-23T01:41:55.2817497Z Entering 'third_party/cudnn_frontend'
2022-11-23T01:41:55.2872208Z Entering 'third_party/cutlass'
2022-11-23T01:41:55.2928626Z Entering 'third_party/eigen'
2022-11-23T01:41:55.2981161Z Entering 'third_party/fbgemm'
2022-11-23T01:41:55.3030432Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:41:55.3078085Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:41:55.3124859Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:41:55.3175074Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:41:55.3227458Z Entering 'third_party/flatbuffers'
2022-11-23T01:41:55.3278916Z Entering 'third_party/fmt'
2022-11-23T01:41:55.3326846Z Entering 'third_party/foxi'
2022-11-23T01:41:55.3374038Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:41:55.3424728Z Entering 'third_party/gloo'
2022-11-23T01:41:55.3473864Z Entering 'third_party/googletest'
2022-11-23T01:41:55.3523233Z Entering 'third_party/ideep'
2022-11-23T01:41:55.3570897Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T01:41:55.3622626Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:41:55.3682499Z Entering 'third_party/ios-cmake'
2022-11-23T01:41:55.3730919Z Entering 'third_party/ittapi'
2022-11-23T01:41:55.3783667Z Entering 'third_party/kineto'
2022-11-23T01:41:55.3830880Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:41:55.3879465Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:41:55.3930403Z Entering 'third_party/nccl/nccl'
2022-11-23T01:41:55.3975064Z Entering 'third_party/neon2sse'
2022-11-23T01:41:55.4025658Z Entering 'third_party/nlohmann'
2022-11-23T01:41:55.4074134Z Entering 'third_party/onnx'
2022-11-23T01:41:55.4136963Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T01:41:55.4187116Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T01:41:55.4241769Z Entering 'third_party/onnx-tensorrt'
2022-11-23T01:41:55.4290799Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:41:55.4344965Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:41:55.4393805Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:41:55.4439664Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:41:55.4494306Z Entering 'third_party/pocketfft'
2022-11-23T01:41:55.4544313Z Entering 'third_party/protobuf'
2022-11-23T01:41:55.4596064Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:41:55.4648746Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T01:41:55.4701108Z Entering 'third_party/psimd'
2022-11-23T01:41:55.4751936Z Entering 'third_party/pthreadpool'
2022-11-23T01:41:55.4801941Z Entering 'third_party/pybind11'
2022-11-23T01:41:55.4850273Z Entering 'third_party/python-enum'
2022-11-23T01:41:55.4901378Z Entering 'third_party/python-peachpy'
2022-11-23T01:41:55.4953855Z Entering 'third_party/python-six'
2022-11-23T01:41:55.5002103Z Entering 'third_party/sleef'
2022-11-23T01:41:55.5048055Z Entering 'third_party/tbb'
2022-11-23T01:41:55.5100364Z Entering 'third_party/tensorpipe'
2022-11-23T01:41:55.5153703Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:41:55.5203076Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:41:55.5249176Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:41:55.5295330Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:41:55.5349312Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:41:55.5399574Z Entering 'third_party/zstd'
2022-11-23T01:41:55.5465868Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url
2022-11-23T01:41:55.5825311Z Entering 'android/libs/fbjni'
2022-11-23T01:41:55.5870250Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2022-11-23T01:41:55.5892171Z Entering 'third_party/FP16'
2022-11-23T01:41:55.5935444Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2022-11-23T01:41:55.5955621Z Entering 'third_party/FXdiv'
2022-11-23T01:41:55.6003700Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2022-11-23T01:41:55.6023224Z Entering 'third_party/NNPACK'
2022-11-23T01:41:55.6074616Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2022-11-23T01:41:55.6092812Z Entering 'third_party/QNNPACK'
2022-11-23T01:41:55.6138970Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config	remote.origin.url
2022-11-23T01:41:55.6160612Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T01:41:55.6205800Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2022-11-23T01:41:55.6226879Z Entering 'third_party/XNNPACK'
2022-11-23T01:41:55.6273679Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2022-11-23T01:41:55.6308845Z Entering 'third_party/benchmark'
2022-11-23T01:41:55.6351937Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2022-11-23T01:41:55.6374212Z Entering 'third_party/cpuinfo'
2022-11-23T01:41:55.6418506Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2022-11-23T01:41:55.6440597Z Entering 'third_party/cub'
2022-11-23T01:41:55.6485905Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config	remote.origin.url
2022-11-23T01:41:55.6507713Z Entering 'third_party/cudnn_frontend'
2022-11-23T01:41:55.6554810Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2022-11-23T01:41:55.6584182Z Entering 'third_party/cutlass'
2022-11-23T01:41:55.6627457Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2022-11-23T01:41:55.6657315Z Entering 'third_party/eigen'
2022-11-23T01:41:55.6706440Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2022-11-23T01:41:55.6729079Z Entering 'third_party/fbgemm'
2022-11-23T01:41:55.6778862Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2022-11-23T01:41:55.6799092Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:41:55.6842268Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2022-11-23T01:41:55.6862476Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:41:55.6905869Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2022-11-23T01:41:55.6925004Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:41:55.6969908Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2022-11-23T01:41:55.6991815Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:41:55.7039672Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config	remote.origin.url
2022-11-23T01:41:55.7059929Z Entering 'third_party/flatbuffers'
2022-11-23T01:41:55.7108304Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2022-11-23T01:41:55.7131604Z Entering 'third_party/fmt'
2022-11-23T01:41:55.7175345Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2022-11-23T01:41:55.7196084Z Entering 'third_party/foxi'
2022-11-23T01:41:55.7240260Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config	remote.origin.url
2022-11-23T01:41:55.7259397Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:41:55.7304786Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2022-11-23T01:41:55.7327609Z Entering 'third_party/gloo'
2022-11-23T01:41:55.7371235Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2022-11-23T01:41:55.7395615Z Entering 'third_party/googletest'
2022-11-23T01:41:55.7442149Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2022-11-23T01:41:55.7462295Z Entering 'third_party/ideep'
2022-11-23T01:41:55.7508639Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2022-11-23T01:41:55.7528212Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T01:41:55.7570694Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2022-11-23T01:41:55.7595184Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:41:55.7642159Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/modules/third_party/oneDNN/config	remote.origin.url
2022-11-23T01:41:55.7673889Z Entering 'third_party/ios-cmake'
2022-11-23T01:41:55.7718655Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config	remote.origin.url
2022-11-23T01:41:55.7736574Z Entering 'third_party/ittapi'
2022-11-23T01:41:55.7781933Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2022-11-23T01:41:55.7804281Z Entering 'third_party/kineto'
2022-11-23T01:41:55.7847924Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2022-11-23T01:41:55.7867759Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:41:55.7909822Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2022-11-23T01:41:55.7931736Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:41:55.7974884Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2022-11-23T01:41:55.7999275Z Entering 'third_party/nccl/nccl'
2022-11-23T01:41:55.8045102Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config	remote.origin.url
2022-11-23T01:41:55.8067470Z Entering 'third_party/neon2sse'
2022-11-23T01:41:55.8113465Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config	remote.origin.url
2022-11-23T01:41:55.8133252Z Entering 'third_party/nlohmann'
2022-11-23T01:41:55.8181026Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2022-11-23T01:41:55.8202094Z Entering 'third_party/onnx'
2022-11-23T01:41:55.8250393Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2022-11-23T01:41:55.8284123Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T01:41:55.8328347Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-11-23T01:41:55.8348451Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T01:41:55.8393850Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-11-23T01:41:55.8414355Z Entering 'third_party/onnx-tensorrt'
2022-11-23T01:41:55.8460024Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config	remote.origin.url
2022-11-23T01:41:55.8480171Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:41:55.8523695Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config	remote.origin.url
2022-11-23T01:41:55.8550249Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:41:55.8597766Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-11-23T01:41:55.8618674Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:41:55.8668030Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-11-23T01:41:55.8686609Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:41:55.8731896Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-11-23T01:41:55.8758865Z Entering 'third_party/pocketfft'
2022-11-23T01:41:55.8803840Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2022-11-23T01:41:55.8824048Z Entering 'third_party/protobuf'
2022-11-23T01:41:55.8868695Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2022-11-23T01:41:55.8893128Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:41:55.8938465Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2022-11-23T01:41:55.8961528Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T01:41:55.9004941Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2022-11-23T01:41:55.9027046Z Entering 'third_party/psimd'
2022-11-23T01:41:55.9071685Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2022-11-23T01:41:55.9091114Z Entering 'third_party/pthreadpool'
2022-11-23T01:41:55.9138041Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2022-11-23T01:41:55.9158095Z Entering 'third_party/pybind11'
2022-11-23T01:41:55.9203386Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2022-11-23T01:41:55.9223498Z Entering 'third_party/python-enum'
2022-11-23T01:41:55.9266324Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config	remote.origin.url
2022-11-23T01:41:55.9289481Z Entering 'third_party/python-peachpy'
2022-11-23T01:41:55.9332735Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2022-11-23T01:41:55.9357288Z Entering 'third_party/python-six'
2022-11-23T01:41:55.9405005Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config	remote.origin.url
2022-11-23T01:41:55.9424407Z Entering 'third_party/sleef'
2022-11-23T01:41:55.9475772Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2022-11-23T01:41:55.9495310Z Entering 'third_party/tbb'
2022-11-23T01:41:55.9541291Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config	remote.origin.url
2022-11-23T01:41:55.9562671Z Entering 'third_party/tensorpipe'
2022-11-23T01:41:55.9608599Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2022-11-23T01:41:55.9629229Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:41:55.9677033Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2022-11-23T01:41:55.9695750Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:41:55.9742420Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2022-11-23T01:41:55.9765940Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:41:55.9806870Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2022-11-23T01:41:55.9828067Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:41:55.9871744Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2022-11-23T01:41:55.9893766Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:41:55.9938910Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-11-23T01:41:55.9961695Z Entering 'third_party/zstd'
2022-11-23T01:41:56.0005863Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config	remote.origin.url
2022-11-23T01:41:56.1100493Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2022-11-23T01:41:56.1465790Z Entering 'android/libs/fbjni'
2022-11-23T01:41:56.1517904Z Entering 'third_party/FP16'
2022-11-23T01:41:56.1569549Z Entering 'third_party/FXdiv'
2022-11-23T01:41:56.1620403Z Entering 'third_party/NNPACK'
2022-11-23T01:41:56.1670263Z Entering 'third_party/QNNPACK'
2022-11-23T01:41:56.1722540Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T01:41:56.1772520Z Entering 'third_party/XNNPACK'
2022-11-23T01:41:56.1837210Z Entering 'third_party/benchmark'
2022-11-23T01:41:56.1888075Z Entering 'third_party/cpuinfo'
2022-11-23T01:41:56.1940902Z Entering 'third_party/cub'
2022-11-23T01:41:56.1994352Z Entering 'third_party/cudnn_frontend'
2022-11-23T01:41:56.2052198Z Entering 'third_party/cutlass'
2022-11-23T01:41:56.2111796Z Entering 'third_party/eigen'
2022-11-23T01:41:56.2164138Z Entering 'third_party/fbgemm'
2022-11-23T01:41:56.2214549Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:41:56.2266710Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:41:56.2315746Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:41:56.2365596Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:41:56.2415176Z Entering 'third_party/flatbuffers'
2022-11-23T01:41:56.2466886Z Entering 'third_party/fmt'
2022-11-23T01:41:56.2514664Z Entering 'third_party/foxi'
2022-11-23T01:41:56.2567380Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:41:56.2613888Z Entering 'third_party/gloo'
2022-11-23T01:41:56.2665517Z Entering 'third_party/googletest'
2022-11-23T01:41:56.2714427Z Entering 'third_party/ideep'
2022-11-23T01:41:56.2764768Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T01:41:56.2814310Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:41:56.2873688Z Entering 'third_party/ios-cmake'
2022-11-23T01:41:56.2924696Z Entering 'third_party/ittapi'
2022-11-23T01:41:56.2973259Z Entering 'third_party/kineto'
2022-11-23T01:41:56.3024098Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:41:56.3073240Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:41:56.3124348Z Entering 'third_party/nccl/nccl'
2022-11-23T01:41:56.3173219Z Entering 'third_party/neon2sse'
2022-11-23T01:41:56.3225527Z Entering 'third_party/nlohmann'
2022-11-23T01:41:56.3274626Z Entering 'third_party/onnx'
2022-11-23T01:41:56.3340113Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T01:41:56.3388847Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T01:41:56.3441545Z Entering 'third_party/onnx-tensorrt'
2022-11-23T01:41:56.3492796Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:41:56.3549082Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:41:56.3598111Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:41:56.3644298Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:41:56.3700488Z Entering 'third_party/pocketfft'
2022-11-23T01:41:56.3751500Z Entering 'third_party/protobuf'
2022-11-23T01:41:56.3807029Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:41:56.3855253Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T01:41:56.3907422Z Entering 'third_party/psimd'
2022-11-23T01:41:56.3954518Z Entering 'third_party/pthreadpool'
2022-11-23T01:41:56.4004497Z Entering 'third_party/pybind11'
2022-11-23T01:41:56.4052749Z Entering 'third_party/python-enum'
2022-11-23T01:41:56.4103155Z Entering 'third_party/python-peachpy'
2022-11-23T01:41:56.4154569Z Entering 'third_party/python-six'
2022-11-23T01:41:56.4205656Z Entering 'third_party/sleef'
2022-11-23T01:41:56.4254686Z Entering 'third_party/tbb'
2022-11-23T01:41:56.4311748Z Entering 'third_party/tensorpipe'
2022-11-23T01:41:56.4360014Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:41:56.4408252Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:41:56.4458138Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:41:56.4508182Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:41:56.4557682Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:41:56.4609283Z Entering 'third_party/zstd'
2022-11-23T01:41:56.4674744Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2022-11-23T01:41:56.5042562Z Entering 'android/libs/fbjni'
2022-11-23T01:41:56.5093957Z Entering 'third_party/FP16'
2022-11-23T01:41:56.5142874Z Entering 'third_party/FXdiv'
2022-11-23T01:41:56.5192299Z Entering 'third_party/NNPACK'
2022-11-23T01:41:56.5244156Z Entering 'third_party/QNNPACK'
2022-11-23T01:41:56.5294155Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T01:41:56.5342785Z Entering 'third_party/XNNPACK'
2022-11-23T01:41:56.5408500Z Entering 'third_party/benchmark'
2022-11-23T01:41:56.5458721Z Entering 'third_party/cpuinfo'
2022-11-23T01:41:56.5511645Z Entering 'third_party/cub'
2022-11-23T01:41:56.5565226Z Entering 'third_party/cudnn_frontend'
2022-11-23T01:41:56.5621611Z Entering 'third_party/cutlass'
2022-11-23T01:41:56.5677671Z Entering 'third_party/eigen'
2022-11-23T01:41:56.5730222Z Entering 'third_party/fbgemm'
2022-11-23T01:41:56.5783855Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:41:56.5832849Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:41:56.5883603Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:41:56.5930602Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:41:56.5980429Z Entering 'third_party/flatbuffers'
2022-11-23T01:41:56.6035869Z Entering 'third_party/fmt'
2022-11-23T01:41:56.6086506Z Entering 'third_party/foxi'
2022-11-23T01:41:56.6138786Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:41:56.6189451Z Entering 'third_party/gloo'
2022-11-23T01:41:56.6239191Z Entering 'third_party/googletest'
2022-11-23T01:41:56.6290535Z Entering 'third_party/ideep'
2022-11-23T01:41:56.6339388Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T01:41:56.6394282Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:41:56.6450266Z Entering 'third_party/ios-cmake'
2022-11-23T01:41:56.6502402Z Entering 'third_party/ittapi'
2022-11-23T01:41:56.6553865Z Entering 'third_party/kineto'
2022-11-23T01:41:56.6604148Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:41:56.6652121Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:41:56.6704528Z Entering 'third_party/nccl/nccl'
2022-11-23T01:41:56.6753385Z Entering 'third_party/neon2sse'
2022-11-23T01:41:56.6804435Z Entering 'third_party/nlohmann'
2022-11-23T01:41:56.6854153Z Entering 'third_party/onnx'
2022-11-23T01:41:56.6918280Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T01:41:56.6970003Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T01:41:56.7021914Z Entering 'third_party/onnx-tensorrt'
2022-11-23T01:41:56.7070010Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:41:56.7126028Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:41:56.7177327Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:41:56.7227537Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:41:56.7286107Z Entering 'third_party/pocketfft'
2022-11-23T01:41:56.7334558Z Entering 'third_party/protobuf'
2022-11-23T01:41:56.7390736Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:41:56.7442355Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T01:41:56.7493765Z Entering 'third_party/psimd'
2022-11-23T01:41:56.7543159Z Entering 'third_party/pthreadpool'
2022-11-23T01:41:56.7594760Z Entering 'third_party/pybind11'
2022-11-23T01:41:56.7646108Z Entering 'third_party/python-enum'
2022-11-23T01:41:56.7696393Z Entering 'third_party/python-peachpy'
2022-11-23T01:41:56.7749720Z Entering 'third_party/python-six'
2022-11-23T01:41:56.7799777Z Entering 'third_party/sleef'
2022-11-23T01:41:56.7849411Z Entering 'third_party/tbb'
2022-11-23T01:41:56.7901989Z Entering 'third_party/tensorpipe'
2022-11-23T01:41:56.7953617Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:41:56.8004815Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:41:56.8055389Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:41:56.8102067Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:41:56.8151933Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:41:56.8204202Z Entering 'third_party/zstd'
2022-11-23T01:41:56.8267645Z ##[endgroup]
2022-11-23T01:41:56.8326235Z [command]/usr/bin/git log -1 --format='%H'
2022-11-23T01:41:56.8361246Z '1cfd3858ac54fe3883534309081631a0a892ba3f'
2022-11-23T01:41:56.8526630Z Prepare all required actions
2022-11-23T01:41:56.8559107Z ##[group]Run ./.github/actions/setup-linux
2022-11-23T01:41:56.8559380Z env:
2022-11-23T01:41:56.8559709Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:41:56.8559918Z ##[endgroup]
2022-11-23T01:41:56.8578409Z ##[group]Run set -euo pipefail
2022-11-23T01:41:56.8578738Z set -euo pipefail
2022-11-23T01:41:56.8579025Z function get_ec2_metadata() {
2022-11-23T01:41:56.8579343Z   # Pulled from instance metadata endpoint for EC2
2022-11-23T01:41:56.8579829Z   # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
2022-11-23T01:41:56.8580236Z   category=$1
2022-11-23T01:41:56.8580549Z   curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
2022-11-23T01:41:56.8580997Z }
2022-11-23T01:41:56.8581272Z echo "ami-id: $(get_ec2_metadata ami-id)"
2022-11-23T01:41:56.8581644Z echo "instance-id: $(get_ec2_metadata instance-id)"
2022-11-23T01:41:56.8582029Z echo "instance-type: $(get_ec2_metadata instance-type)"
2022-11-23T01:41:56.8582378Z echo "system info $(uname -a)"
2022-11-23T01:41:56.8597075Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:41:56.8597372Z env:
2022-11-23T01:41:56.8597618Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:41:56.8597857Z ##[endgroup]
2022-11-23T01:41:56.8712339Z ami-id: ami-096198a0bccc6bad4
2022-11-23T01:41:56.8778073Z instance-id: i-018c617fc63e71812
2022-11-23T01:41:56.8854971Z instance-type: g3.16xlarge
2022-11-23T01:41:56.8865280Z system info Linux ip-10-0-2-152.ec2.internal 4.14.252-195.483.amzn2.x86_64 #1 SMP Mon Nov 1 20:58:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
2022-11-23T01:41:56.8883827Z ##[group]Run if systemctl is-active --quiet docker; then
2022-11-23T01:41:56.8884311Z if systemctl is-active --quiet docker; then
2022-11-23T01:41:56.8884618Z     echo "Docker daemon is running...";
2022-11-23T01:41:56.8884935Z else
2022-11-23T01:41:56.8885158Z     echo "Starting docker deamon..." && sudo systemctl start docker;
2022-11-23T01:41:56.8885473Z fi
2022-11-23T01:41:56.8897773Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:41:56.8898027Z env:
2022-11-23T01:41:56.8898271Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:41:56.8898516Z ##[endgroup]
2022-11-23T01:41:56.8954035Z Docker daemon is running...
2022-11-23T01:41:56.8974907Z ##[group]Run AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-11-23T01:41:56.8975411Z AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-11-23T01:41:56.8975817Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-11-23T01:41:56.8976371Z retry aws ecr get-login*** "$AWS_DEFAULT_REGION" | docker login --username AWS \
2022-11-23T01:41:56.8977179Z     --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
2022-11-23T01:41:56.8990630Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:41:56.8990942Z env:
2022-11-23T01:41:56.8991194Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:41:56.8991452Z   AWS_RETRY_MODE: standard
2022-11-23T01:41:56.8991717Z   AWS_MAX_ATTEMPTS: 5
2022-11-23T01:41:56.8991995Z   AWS_DEFAULT_REGION: us-east-1
2022-11-23T01:41:56.8992242Z ##[endgroup]
2022-11-23T01:41:57.8783451Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2022-11-23T01:41:57.8783960Z Configure a credential helper to remove this warning. See
2022-11-23T01:41:57.8784519Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2022-11-23T01:41:57.8784810Z 
2022-11-23T01:41:57.8785467Z Login Succeeded
2022-11-23T01:41:57.8871435Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-11-23T01:41:57.8871851Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-11-23T01:41:57.8872319Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-11-23T01:41:57.8885475Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:41:57.8885774Z env:
2022-11-23T01:41:57.8885996Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:41:57.8886256Z ##[endgroup]
2022-11-23T01:41:57.8985545Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main
2022-11-23T01:41:57.8985851Z with:
2022-11-23T01:41:57.8986347Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:41:57.8986820Z env:
2022-11-23T01:41:57.8987074Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:41:57.8987475Z ##[endgroup]
2022-11-23T01:41:57.9004098Z ##[group]Run retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-11-23T01:41:57.9004398Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-11-23T01:41:57.9004882Z # ignore output since only exit code is used for conditional
2022-11-23T01:41:57.9005171Z # only pull docker image if it's not available locally
2022-11-23T01:41:57.9005569Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2022-11-23T01:41:57.9005985Z   retry docker pull "${DOCKER_IMAGE}"
2022-11-23T01:41:57.9006271Z fi
2022-11-23T01:41:57.9020301Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:41:57.9020615Z env:
2022-11-23T01:41:57.9020867Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:41:57.9021368Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:41:57.9021849Z ##[endgroup]
2022-11-23T01:41:58.1310767Z 072aae4a77ed7d3a69ad5683420509c41301b940: Pulling from pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7
2022-11-23T01:41:58.1311235Z a404e5416296: Pulling fs layer
2022-11-23T01:41:58.1311548Z c58c079e9b17: Pulling fs layer
2022-11-23T01:41:58.1311764Z e5b80b8bbe91: Pulling fs layer
2022-11-23T01:41:58.1312106Z 888240790290: Pulling fs layer
2022-11-23T01:41:58.1312538Z 515fe5e34eb4: Pulling fs layer
2022-11-23T01:41:58.1312811Z 4e4521f12f5a: Pulling fs layer
2022-11-23T01:41:58.1313130Z f6e1a56cb32d: Pulling fs layer
2022-11-23T01:41:58.1313423Z c29b96e36bd0: Pulling fs layer
2022-11-23T01:41:58.1316547Z 304d3c6c28d0: Pulling fs layer
2022-11-23T01:41:58.1316979Z fac00e927cfe: Pulling fs layer
2022-11-23T01:41:58.1317265Z f0158c8d8420: Pulling fs layer
2022-11-23T01:41:58.1317572Z 3ceac802dd07: Pulling fs layer
2022-11-23T01:41:58.1317845Z 0d0e625ba887: Pulling fs layer
2022-11-23T01:41:58.1318130Z bc2be817cb7e: Pulling fs layer
2022-11-23T01:41:58.1318489Z 888240790290: Waiting
2022-11-23T01:41:58.1318950Z 11eb2106b948: Pulling fs layer
2022-11-23T01:41:58.1319340Z 304d3c6c28d0: Waiting
2022-11-23T01:41:58.1319680Z 515fe5e34eb4: Waiting
2022-11-23T01:41:58.1320187Z 34fa4193c7a6: Pulling fs layer
2022-11-23T01:41:58.1320668Z f6e1a56cb32d: Waiting
2022-11-23T01:41:58.1321070Z a7cf5b3894f8: Pulling fs layer
2022-11-23T01:41:58.1321608Z 3ceac802dd07: Waiting
2022-11-23T01:41:58.1322055Z 3f6b06edd3f5: Pulling fs layer
2022-11-23T01:41:58.1322344Z 0d0e625ba887: Waiting
2022-11-23T01:41:58.1322578Z f0158c8d8420: Waiting
2022-11-23T01:41:58.1322829Z c29b96e36bd0: Waiting
2022-11-23T01:41:58.1323254Z 73a2b1f75a3d: Pulling fs layer
2022-11-23T01:41:58.1323769Z ba6235196410: Pulling fs layer
2022-11-23T01:41:58.1324147Z 11eb2106b948: Waiting
2022-11-23T01:41:58.1324658Z bc2be817cb7e: Waiting
2022-11-23T01:41:58.1325152Z 879cdaf83543: Pulling fs layer
2022-11-23T01:41:58.1325557Z a7cf5b3894f8: Waiting
2022-11-23T01:41:58.1325951Z 6ff0fc00b0a9: Pulling fs layer
2022-11-23T01:41:58.1326445Z a58b9ed071f4: Pulling fs layer
2022-11-23T01:41:58.1326852Z 3f6b06edd3f5: Waiting
2022-11-23T01:41:58.1327188Z a8c562f6a1cf: Pulling fs layer
2022-11-23T01:41:58.1327389Z 879cdaf83543: Waiting
2022-11-23T01:41:58.1327611Z 4e4521f12f5a: Waiting
2022-11-23T01:41:58.1327871Z 0a39b4492650: Pulling fs layer
2022-11-23T01:41:58.1328137Z a58b9ed071f4: Waiting
2022-11-23T01:41:58.1328356Z ba6235196410: Waiting
2022-11-23T01:41:58.1328704Z 9088ff8de269: Pulling fs layer
2022-11-23T01:41:58.1328988Z 73a2b1f75a3d: Waiting
2022-11-23T01:41:58.1329408Z 165006759af3: Pulling fs layer
2022-11-23T01:41:58.1329720Z ae48b7377a0d: Pulling fs layer
2022-11-23T01:41:58.1330033Z b18965f4b6f1: Pulling fs layer
2022-11-23T01:41:58.1330289Z 102ddcd90753: Pulling fs layer
2022-11-23T01:41:58.1330600Z 5f5dd1cba120: Pulling fs layer
2022-11-23T01:41:58.1330808Z b18965f4b6f1: Waiting
2022-11-23T01:41:58.1331063Z 8a7f50c8b503: Pulling fs layer
2022-11-23T01:41:58.1331330Z 863c35620b44: Pulling fs layer
2022-11-23T01:41:58.1331582Z 102ddcd90753: Waiting
2022-11-23T01:41:58.1332032Z ae48b7377a0d: Waiting
2022-11-23T01:41:58.1332191Z 183e4209dc37: Pulling fs layer
2022-11-23T01:41:58.1332499Z a47cba6c334e: Pulling fs layer
2022-11-23T01:41:58.1332739Z 8a7f50c8b503: Waiting
2022-11-23T01:41:58.1333077Z a9f3d4742233: Pulling fs layer
2022-11-23T01:41:58.1333346Z 3cefa8a4607f: Pulling fs layer
2022-11-23T01:41:58.1333527Z 023a41fa48e6: Pulling fs layer
2022-11-23T01:41:58.1333787Z 96e251412f4d: Pulling fs layer
2022-11-23T01:41:58.1334171Z 49d40c00cf56: Pulling fs layer
2022-11-23T01:41:58.1334414Z 5f5dd1cba120: Waiting
2022-11-23T01:41:58.1334682Z a47cba6c334e: Waiting
2022-11-23T01:41:58.1334853Z 7e2d6313145f: Pulling fs layer
2022-11-23T01:41:58.1335111Z 863c35620b44: Waiting
2022-11-23T01:41:58.1335328Z 023a41fa48e6: Waiting
2022-11-23T01:41:58.1335589Z 96805775a692: Pulling fs layer
2022-11-23T01:41:58.1335853Z 183e4209dc37: Waiting
2022-11-23T01:41:58.1336157Z 96e251412f4d: Waiting
2022-11-23T01:41:58.1336449Z 3cefa8a4607f: Waiting
2022-11-23T01:41:58.1336677Z 49d40c00cf56: Waiting
2022-11-23T01:41:58.1337221Z a9f3d4742233: Waiting
2022-11-23T01:41:58.1337511Z 75f1ead35ace: Pulling fs layer
2022-11-23T01:41:58.1337732Z 793c37004dab: Pulling fs layer
2022-11-23T01:41:58.1337937Z cadc5661750d: Pulling fs layer
2022-11-23T01:41:58.1338316Z 75f1ead35ace: Waiting
2022-11-23T01:41:58.1338497Z 6386b2adbe28: Pulling fs layer
2022-11-23T01:41:58.1338850Z cadc5661750d: Waiting
2022-11-23T01:41:58.1339055Z 7e2d6313145f: Waiting
2022-11-23T01:41:58.1339231Z 793c37004dab: Waiting
2022-11-23T01:41:58.1339576Z 74aa250bc82f: Pulling fs layer
2022-11-23T01:41:58.1339880Z 436525efe61d: Pulling fs layer
2022-11-23T01:41:58.1340073Z 596be1fe0bda: Pulling fs layer
2022-11-23T01:41:58.1340343Z 772fa4efddc3: Pulling fs layer
2022-11-23T01:41:58.1340588Z 74aa250bc82f: Waiting
2022-11-23T01:41:58.1340860Z 91ddf385377b: Pulling fs layer
2022-11-23T01:41:58.1341242Z 9f7cfb895784: Pulling fs layer
2022-11-23T01:41:58.1341484Z 436525efe61d: Waiting
2022-11-23T01:41:58.1341652Z 772fa4efddc3: Waiting
2022-11-23T01:41:58.1341916Z 8b8218af0479: Pulling fs layer
2022-11-23T01:41:58.1342252Z 596be1fe0bda: Waiting
2022-11-23T01:41:58.1342518Z 8b8218af0479: Waiting
2022-11-23T01:41:58.1342671Z 91ddf385377b: Waiting
2022-11-23T01:41:58.2528663Z c58c079e9b17: Verifying Checksum
2022-11-23T01:41:58.2529184Z c58c079e9b17: Download complete
2022-11-23T01:41:58.3512343Z 888240790290: Verifying Checksum
2022-11-23T01:41:58.3512712Z 888240790290: Download complete
2022-11-23T01:41:58.4076757Z e5b80b8bbe91: Verifying Checksum
2022-11-23T01:41:58.4077061Z e5b80b8bbe91: Download complete
2022-11-23T01:41:58.4456175Z 515fe5e34eb4: Verifying Checksum
2022-11-23T01:41:58.4456449Z 515fe5e34eb4: Download complete
2022-11-23T01:41:58.4466698Z a404e5416296: Verifying Checksum
2022-11-23T01:41:58.4466965Z a404e5416296: Download complete
2022-11-23T01:41:58.5258306Z f6e1a56cb32d: Download complete
2022-11-23T01:41:58.6175140Z 304d3c6c28d0: Download complete
2022-11-23T01:41:58.7064394Z fac00e927cfe: Verifying Checksum
2022-11-23T01:41:58.7064763Z fac00e927cfe: Download complete
2022-11-23T01:41:59.1786074Z a404e5416296: Pull complete
2022-11-23T01:41:59.4442177Z c58c079e9b17: Pull complete
2022-11-23T01:41:59.9728801Z e5b80b8bbe91: Pull complete
2022-11-23T01:42:00.1082406Z 888240790290: Pull complete
2022-11-23T01:42:00.2211292Z 515fe5e34eb4: Pull complete
2022-11-23T01:42:00.7588707Z f0158c8d8420: Verifying Checksum
2022-11-23T01:42:00.7589048Z f0158c8d8420: Download complete
2022-11-23T01:42:00.8371581Z 3ceac802dd07: Verifying Checksum
2022-11-23T01:42:00.8372331Z 3ceac802dd07: Download complete
2022-11-23T01:42:00.9378216Z 0d0e625ba887: Verifying Checksum
2022-11-23T01:42:00.9378534Z 0d0e625ba887: Download complete
2022-11-23T01:42:01.0267001Z bc2be817cb7e: Verifying Checksum
2022-11-23T01:42:01.0267357Z bc2be817cb7e: Download complete
2022-11-23T01:42:01.7617571Z 11eb2106b948: Verifying Checksum
2022-11-23T01:42:01.7617944Z 11eb2106b948: Download complete
2022-11-23T01:42:01.8441569Z 34fa4193c7a6: Verifying Checksum
2022-11-23T01:42:01.8442299Z 34fa4193c7a6: Download complete
2022-11-23T01:42:01.9337597Z a7cf5b3894f8: Download complete
2022-11-23T01:42:09.6524048Z 4e4521f12f5a: Verifying Checksum
2022-11-23T01:42:09.6525021Z 4e4521f12f5a: Download complete
2022-11-23T01:42:09.7391344Z 73a2b1f75a3d: Verifying Checksum
2022-11-23T01:42:09.7391714Z 73a2b1f75a3d: Download complete
2022-11-23T01:42:09.8249777Z ba6235196410: Verifying Checksum
2022-11-23T01:42:09.8250154Z ba6235196410: Download complete
2022-11-23T01:42:09.9155226Z 879cdaf83543: Download complete
2022-11-23T01:42:09.9924297Z 6ff0fc00b0a9: Download complete
2022-11-23T01:42:10.0926822Z a58b9ed071f4: Verifying Checksum
2022-11-23T01:42:10.0927088Z a58b9ed071f4: Download complete
2022-11-23T01:42:10.1732360Z a8c562f6a1cf: Verifying Checksum
2022-11-23T01:42:10.1732693Z a8c562f6a1cf: Download complete
2022-11-23T01:42:11.1132756Z 0a39b4492650: Verifying Checksum
2022-11-23T01:42:11.1133067Z 0a39b4492650: Download complete
2022-11-23T01:42:11.1837051Z 9088ff8de269: Download complete
2022-11-23T01:42:11.2529646Z 165006759af3: Verifying Checksum
2022-11-23T01:42:11.2529982Z 165006759af3: Download complete
2022-11-23T01:42:11.3425555Z ae48b7377a0d: Verifying Checksum
2022-11-23T01:42:11.3425944Z ae48b7377a0d: Download complete
2022-11-23T01:42:11.4281041Z b18965f4b6f1: Verifying Checksum
2022-11-23T01:42:11.4281359Z b18965f4b6f1: Download complete
2022-11-23T01:42:11.5088690Z 102ddcd90753: Verifying Checksum
2022-11-23T01:42:11.5089327Z 102ddcd90753: Download complete
2022-11-23T01:42:12.8020194Z c29b96e36bd0: Verifying Checksum
2022-11-23T01:42:12.8020547Z c29b96e36bd0: Download complete
2022-11-23T01:42:12.8811070Z 8a7f50c8b503: Verifying Checksum
2022-11-23T01:42:12.8811418Z 8a7f50c8b503: Download complete
2022-11-23T01:42:12.9606281Z 863c35620b44: Verifying Checksum
2022-11-23T01:42:13.3632254Z 183e4209dc37: Verifying Checksum
2022-11-23T01:42:13.3632883Z 183e4209dc37: Download complete
2022-11-23T01:42:13.4853118Z a47cba6c334e: Verifying Checksum
2022-11-23T01:42:13.4853745Z a47cba6c334e: Download complete
2022-11-23T01:42:13.5045519Z 5f5dd1cba120: Verifying Checksum
2022-11-23T01:42:13.5046275Z 5f5dd1cba120: Download complete
2022-11-23T01:42:13.5642314Z a9f3d4742233: Download complete
2022-11-23T01:42:13.6598148Z 023a41fa48e6: Download complete
2022-11-23T01:42:13.7512710Z 3cefa8a4607f: Verifying Checksum
2022-11-23T01:42:13.7513063Z 3cefa8a4607f: Download complete
2022-11-23T01:42:13.8202411Z 49d40c00cf56: Verifying Checksum
2022-11-23T01:42:13.8203018Z 49d40c00cf56: Download complete
2022-11-23T01:42:13.9247625Z 7e2d6313145f: Verifying Checksum
2022-11-23T01:42:13.9247976Z 7e2d6313145f: Download complete
2022-11-23T01:42:14.1010835Z 96e251412f4d: Verifying Checksum
2022-11-23T01:42:14.1011196Z 96e251412f4d: Download complete
2022-11-23T01:42:14.1708013Z 75f1ead35ace: Verifying Checksum
2022-11-23T01:42:14.1708435Z 75f1ead35ace: Download complete
2022-11-23T01:42:14.2642509Z 793c37004dab: Verifying Checksum
2022-11-23T01:42:14.2642880Z 793c37004dab: Download complete
2022-11-23T01:42:14.3417222Z cadc5661750d: Download complete
2022-11-23T01:42:14.4095030Z 6386b2adbe28: Download complete
2022-11-23T01:42:14.6024038Z 74aa250bc82f: Verifying Checksum
2022-11-23T01:42:14.6024397Z 74aa250bc82f: Download complete
2022-11-23T01:42:14.6794219Z 436525efe61d: Download complete
2022-11-23T01:42:15.2873597Z 596be1fe0bda: Verifying Checksum
2022-11-23T01:42:15.2873953Z 596be1fe0bda: Download complete
2022-11-23T01:42:15.3578539Z 772fa4efddc3: Verifying Checksum
2022-11-23T01:42:15.3579146Z 772fa4efddc3: Download complete
2022-11-23T01:42:18.5738008Z 96805775a692: Verifying Checksum
2022-11-23T01:42:18.5738360Z 96805775a692: Download complete
2022-11-23T01:42:18.6863216Z 9f7cfb895784: Verifying Checksum
2022-11-23T01:42:18.6863588Z 9f7cfb895784: Download complete
2022-11-23T01:42:18.7633184Z 8b8218af0479: Verifying Checksum
2022-11-23T01:42:18.7633525Z 8b8218af0479: Download complete
2022-11-23T01:42:21.7299806Z 3f6b06edd3f5: Verifying Checksum
2022-11-23T01:42:21.7300169Z 3f6b06edd3f5: Download complete
2022-11-23T01:42:22.9707325Z 4e4521f12f5a: Pull complete
2022-11-23T01:42:23.1050729Z f6e1a56cb32d: Pull complete
2022-11-23T01:42:42.9204943Z c29b96e36bd0: Pull complete
2022-11-23T01:42:44.6322180Z 304d3c6c28d0: Pull complete
2022-11-23T01:42:45.3543826Z 91ddf385377b: Verifying Checksum
2022-11-23T01:42:45.3544173Z 91ddf385377b: Download complete
2022-11-23T01:42:46.5400172Z fac00e927cfe: Pull complete
2022-11-23T01:42:54.2428881Z f0158c8d8420: Pull complete
2022-11-23T01:42:56.1229011Z 3ceac802dd07: Pull complete
2022-11-23T01:42:57.9702789Z 0d0e625ba887: Pull complete
2022-11-23T01:42:59.8529177Z bc2be817cb7e: Pull complete
2022-11-23T01:43:03.7013000Z 11eb2106b948: Pull complete
2022-11-23T01:43:05.5496175Z 34fa4193c7a6: Pull complete
2022-11-23T01:43:07.5561853Z a7cf5b3894f8: Pull complete
2022-11-23T01:43:40.6929727Z 3f6b06edd3f5: Pull complete
2022-11-23T01:43:42.5693743Z 73a2b1f75a3d: Pull complete
2022-11-23T01:43:43.8588459Z ba6235196410: Pull complete
2022-11-23T01:43:43.9663504Z 879cdaf83543: Pull complete
2022-11-23T01:43:44.0811088Z 6ff0fc00b0a9: Pull complete
2022-11-23T01:43:44.1782349Z a58b9ed071f4: Pull complete
2022-11-23T01:43:44.2805311Z a8c562f6a1cf: Pull complete
2022-11-23T01:43:46.3362595Z 0a39b4492650: Pull complete
2022-11-23T01:43:46.4469111Z 9088ff8de269: Pull complete
2022-11-23T01:43:46.5687728Z 165006759af3: Pull complete
2022-11-23T01:43:46.7145595Z ae48b7377a0d: Pull complete
2022-11-23T01:43:46.8343290Z b18965f4b6f1: Pull complete
2022-11-23T01:43:46.9444514Z 102ddcd90753: Pull complete
2022-11-23T01:43:54.4484149Z 5f5dd1cba120: Pull complete
2022-11-23T01:43:56.3669799Z 8a7f50c8b503: Pull complete
2022-11-23T01:43:58.2418092Z 863c35620b44: Pull complete
2022-11-23T01:44:00.9935346Z 183e4209dc37: Pull complete
2022-11-23T01:44:02.8676227Z a47cba6c334e: Pull complete
2022-11-23T01:44:05.2803168Z a9f3d4742233: Pull complete
2022-11-23T01:44:08.9992710Z 3cefa8a4607f: Pull complete
2022-11-23T01:44:11.4047300Z 023a41fa48e6: Pull complete
2022-11-23T01:44:15.3844292Z 96e251412f4d: Pull complete
2022-11-23T01:44:18.3911118Z 49d40c00cf56: Pull complete
2022-11-23T01:44:20.8136334Z 7e2d6313145f: Pull complete
2022-11-23T01:44:28.4920628Z 96805775a692: Pull complete
2022-11-23T01:44:31.4107389Z 75f1ead35ace: Pull complete
2022-11-23T01:44:33.1738177Z 793c37004dab: Pull complete
2022-11-23T01:44:35.0828534Z cadc5661750d: Pull complete
2022-11-23T01:44:37.0343296Z 6386b2adbe28: Pull complete
2022-11-23T01:44:40.3480221Z 74aa250bc82f: Pull complete
2022-11-23T01:44:42.1945749Z 436525efe61d: Pull complete
2022-11-23T01:44:46.1894440Z 596be1fe0bda: Pull complete
2022-11-23T01:44:47.2922947Z 772fa4efddc3: Pull complete
2022-11-23T01:45:27.3331549Z 91ddf385377b: Pull complete
2022-11-23T01:45:29.2750609Z 9f7cfb895784: Pull complete
2022-11-23T01:45:31.1218443Z 8b8218af0479: Pull complete
2022-11-23T01:45:32.4696444Z Digest: sha256:3a5626edfb2c43fb24303351be75287af92426b6bb7c6df2defc98f980346c6a
2022-11-23T01:45:33.1104215Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:45:33.2222035Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:45:33.2338412Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main
2022-11-23T01:45:33.2338754Z with:
2022-11-23T01:45:33.2339016Z   driver-version: 515.76
2022-11-23T01:45:33.2339269Z env:
2022-11-23T01:45:33.2339496Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:45:33.2339766Z ##[endgroup]
2022-11-23T01:45:33.3785657Z ##[group]Run nick-fields/retry@7d4a37704547a311dbb66ebdf5b23ec19374a767
2022-11-23T01:45:33.3786157Z with:
2022-11-23T01:45:33.3786384Z   timeout_minutes: 10
2022-11-23T01:45:33.3786654Z   max_attempts: 3
2022-11-23T01:45:33.3792938Z   command: # Is it disgusting to have a full shell script here in this github action? Sure
# But is it the best way to make it so that this action relies on nothing else? Absolutely
set -eou pipefail

DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID)
DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run"
YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo"

install_nvidia_docker2_amzn2() {
    (
        set -x
        # Needed for yum-config-manager
        sudo yum install -y yum-utils
        sudo yum-config-manager --add-repo "${YUM_REPO_URL}"
        sudo yum install -y nvidia-docker2
        sudo systemctl restart docker
    )
}

install_nvidia_driver_amzn2() {
    (
        set -x

        # Purge any nvidia driver installed from RHEL repo
        sudo yum remove -y nvidia-driver-latest-dkms

        # Try to gather more information about the runner and its existing NVIDIA driver if any
        echo "Before installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        HAS_NVIDIA_DRIVER=0
        # Check if NVIDIA driver has already been installed
        if [ -x "$(command -v nvidia-smi)" ]; then
            set +e
            # The driver exists, check its version next. Also check only the first GPU if there are more than one of them
            # so that the same driver version is not print over multiple lines
            INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
            NVIDIA_SMI_STATUS=$?

            if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing"
            elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing"
            else
                HAS_NVIDIA_DRIVER=1
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation"
            fi
            set -e
        fi

        if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then
            sudo yum groupinstall -y "Development Tools"
            # ensure our kernel install is the same as our underlying kernel,
            # groupinstall "Development Tools" has a habit of mismatching kernel headers
            sudo yum install -y "kernel-devel-uname-r == $(uname -r)"
            sudo modprobe backlight
            sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN"

            set +e
            sudo /bin/bash /tmp/nvidia_driver -s --no-drm
            NVIDIA_INSTALLATION_STATUS=$?

            RESET_GPU=0
            if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then
                sudo cat /var/log/nvidia-installer.log
                # Fail to install NVIDIA driver, try to reset the GPU
                RESET_GPU=1
            elif [ -x "$(command -v nvidia-smi)" ]; then
                # Check again if nvidia-smi works even if the driver installation completes successfully
                INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
                NVIDIA_SMI_STATUS=$?

                if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                    RESET_GPU=1
                fi
            fi

            if [ "$RESET_GPU" -eq 1 ]; then
                NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1)
                # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this
                # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388
                for PCI_ID in $NVIDIA_DEVICES; do
                    DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable)

                    echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)"
                    # This requires sudo permission of course
                    echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset
                    sleep 1
                done
            fi

            sudo rm -fv /tmp/nvidia_driver
            set -e
        fi

        sudo modprobe nvidia || true
        echo "After installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        (
            set +e
            nvidia-smi
            NVIDIA_SMI_STATUS=$?

            # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285
            if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then
                echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}"
            else
                echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}"
                exit ${NVIDIA_SMI_STATUS}
            fi
            set -e
        )
    )
}

echo "== Installing nvidia driver ${DRIVER_FN} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_driver_amzn2
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac

# Install container toolkit based on distribution
echo "== Installing nvidia container toolkit for ${DISTRIBUTION} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_docker2_amzn2
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac
echo "GPU_FLAG=--gpus all" >> "${GITHUB_ENV}"

2022-11-23T01:45:33.3799222Z   retry_wait_seconds: 10
2022-11-23T01:45:33.3799520Z   polling_interval_seconds: 1
2022-11-23T01:45:33.3799814Z   warning_on_retry: true
2022-11-23T01:45:33.3800397Z   continue_on_error: false
2022-11-23T01:45:33.3800663Z env:
2022-11-23T01:45:33.3800920Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:45:33.3801182Z   DRIVER_VERSION: 515.76
2022-11-23T01:45:33.3801451Z ##[endgroup]
2022-11-23T01:45:33.4391664Z 
2022-11-23T01:45:33.4414801Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-11-23T01:45:33.4463140Z == Installing nvidia driver NVIDIA-Linux-x86_64-515.76.run ==
2022-11-23T01:45:33.4464326Z + sudo yum remove -y nvidia-driver-latest-dkms
2022-11-23T01:45:33.9414834Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-11-23T01:45:33.9863011Z No Match for argument: nvidia-driver-latest-dkms
2022-11-23T01:45:34.0160900Z No Packages marked for removal
2022-11-23T01:45:34.0334100Z + echo 'Before installing NVIDIA driver'
2022-11-23T01:45:34.0334456Z + lspci
2022-11-23T01:45:34.0334736Z Before installing NVIDIA driver
2022-11-23T01:45:35.1729683Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
2022-11-23T01:45:35.1730197Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2022-11-23T01:45:35.1730693Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
2022-11-23T01:45:35.1730989Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01)
2022-11-23T01:45:35.1731397Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446
2022-11-23T01:45:35.1731822Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2022-11-23T01:45:35.1732549Z 00:1b.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2022-11-23T01:45:35.1733105Z 00:1c.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2022-11-23T01:45:35.1733528Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2022-11-23T01:45:35.1733988Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2022-11-23T01:45:35.1734298Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01)
2022-11-23T01:45:35.1738767Z + lsmod
2022-11-23T01:45:35.1756573Z Module                  Size  Used by
2022-11-23T01:45:35.1756865Z xt_conntrack           16384  1
2022-11-23T01:45:35.1757177Z ipt_MASQUERADE         16384  1
2022-11-23T01:45:35.1757496Z nf_nat_masquerade_ipv4    16384  1 ipt_MASQUERADE
2022-11-23T01:45:35.1757990Z nf_conntrack_netlink    49152  0
2022-11-23T01:45:35.1758252Z nfnetlink              16384  2 nf_conntrack_netlink
2022-11-23T01:45:35.1758534Z xfrm_user              45056  1
2022-11-23T01:45:35.1758833Z xfrm_algo              16384  1 xfrm_user
2022-11-23T01:45:35.1759184Z xt_addrtype            16384  2
2022-11-23T01:45:35.1759383Z iptable_filter         16384  1
2022-11-23T01:45:35.1759725Z iptable_nat            16384  1
2022-11-23T01:45:35.1759966Z nf_conntrack_ipv4      16384  3
2022-11-23T01:45:35.1762137Z nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
2022-11-23T01:45:35.1762446Z nf_nat_ipv4            16384  1 iptable_nat
2022-11-23T01:45:35.1762839Z nf_nat                 36864  2 nf_nat_masquerade_ipv4,nf_nat_ipv4
2022-11-23T01:45:35.1763331Z nf_conntrack          155648  7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink
2022-11-23T01:45:35.1763731Z br_netfilter           24576  0
2022-11-23T01:45:35.1764028Z bridge                172032  1 br_netfilter
2022-11-23T01:45:35.1764327Z stp                    16384  1 bridge
2022-11-23T01:45:35.1764636Z llc                    16384  2 bridge,stp
2022-11-23T01:45:35.1764892Z overlay                86016  0
2022-11-23T01:45:35.1765167Z sunrpc                393216  1
2022-11-23T01:45:35.1765441Z dm_mirror              28672  0
2022-11-23T01:45:35.1765703Z dm_region_hash         20480  1 dm_mirror
2022-11-23T01:45:35.1766027Z dm_log                 20480  2 dm_region_hash,dm_mirror
2022-11-23T01:45:35.1766242Z dm_mod                143360  2 dm_log,dm_mirror
2022-11-23T01:45:35.1766535Z dax                    69632  1 dm_mod
2022-11-23T01:45:35.1766774Z sb_edac                24576  0
2022-11-23T01:45:35.1767045Z crc32_pclmul           16384  0
2022-11-23T01:45:35.1767309Z ghash_clmulni_intel    16384  0
2022-11-23T01:45:35.1767676Z pcbc                   16384  0
2022-11-23T01:45:35.1767944Z aesni_intel           188416  0
2022-11-23T01:45:35.1768192Z ata_piix               36864  0
2022-11-23T01:45:35.1768467Z aes_x86_64             20480  1 aesni_intel
2022-11-23T01:45:35.1768760Z crypto_simd            16384  1 aesni_intel
2022-11-23T01:45:35.1768943Z glue_helper            16384  1 aesni_intel
2022-11-23T01:45:35.1769234Z pcc_cpufreq            16384  0
2022-11-23T01:45:35.1769550Z libata                266240  1 ata_piix
2022-11-23T01:45:35.1769863Z cryptd                 28672  3 crypto_simd,ghash_clmulni_intel,aesni_intel
2022-11-23T01:45:35.1770174Z mousedev               24576  0
2022-11-23T01:45:35.1770439Z evdev                  20480  3
2022-11-23T01:45:35.1770793Z scsi_mod              245760  1 libata
2022-11-23T01:45:35.1770966Z psmouse                32768  0
2022-11-23T01:45:35.1771220Z button                 16384  0
2022-11-23T01:45:35.1771469Z ena                   114688  0
2022-11-23T01:45:35.1771707Z xen_blkfront           49152  2
2022-11-23T01:45:35.1772071Z crc32c_intel           24576  0
2022-11-23T01:45:35.1772290Z autofs4                49152  2
2022-11-23T01:45:35.1772547Z + modinfo nvidia
2022-11-23T01:45:35.1772836Z modinfo: ERROR: Module nvidia not found.
2022-11-23T01:45:35.1773110Z + true
2022-11-23T01:45:35.1773408Z + HAS_NVIDIA_DRIVER=0
2022-11-23T01:45:35.1774556Z ++ command -v nvidia-smi
2022-11-23T01:45:35.1775547Z + '[' -x '' ']'
2022-11-23T01:45:35.1776084Z + '[' 0 -eq 0 ']'
2022-11-23T01:45:35.1777443Z + sudo yum groupinstall -y 'Development Tools'
2022-11-23T01:45:35.6616275Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-11-23T01:45:35.9562032Z Resolving Dependencies
2022-11-23T01:45:35.9566246Z --> Running transaction check
2022-11-23T01:45:35.9567473Z ---> Package autoconf.noarch 0:2.69-11.amzn2 will be installed
2022-11-23T01:45:35.9787658Z --> Processing Dependency: m4 >= 1.4.14 for package: autoconf-2.69-11.amzn2.noarch
2022-11-23T01:45:36.1859329Z --> Processing Dependency: perl(Data::Dumper) for package: autoconf-2.69-11.amzn2.noarch
2022-11-23T01:45:36.1860173Z ---> Package automake.noarch 0:1.13.4-3.1.amzn2 will be installed
2022-11-23T01:45:36.1904755Z --> Processing Dependency: perl(Thread::Queue) for package: automake-1.13.4-3.1.amzn2.noarch
2022-11-23T01:45:36.1911519Z --> Processing Dependency: perl(TAP::Parser) for package: automake-1.13.4-3.1.amzn2.noarch
2022-11-23T01:45:36.1921994Z ---> Package bison.x86_64 0:3.0.4-6.amzn2.0.2 will be installed
2022-11-23T01:45:36.2038605Z ---> Package byacc.x86_64 0:1.9.20130304-3.amzn2.0.2 will be installed
2022-11-23T01:45:36.2045204Z ---> Package cscope.x86_64 0:15.8-10.amzn2.0.2 will be installed
2022-11-23T01:45:36.2089647Z --> Processing Dependency: emacs-filesystem for package: cscope-15.8-10.amzn2.0.2.x86_64
2022-11-23T01:45:36.2113010Z ---> Package ctags.x86_64 0:5.8-13.amzn2.0.2 will be installed
2022-11-23T01:45:36.2121658Z ---> Package diffstat.x86_64 0:1.57-4.amzn2.0.2 will be installed
2022-11-23T01:45:36.2129498Z ---> Package doxygen.x86_64 1:1.8.5-4.amzn2 will be installed
2022-11-23T01:45:36.2230616Z ---> Package elfutils.x86_64 0:0.176-2.amzn2 will be installed
2022-11-23T01:45:36.2366724Z ---> Package flex.x86_64 0:2.5.37-3.amzn2.0.3 will be installed
2022-11-23T01:45:36.2385622Z ---> Package gcc.x86_64 0:7.3.1-15.amzn2 will be installed
2022-11-23T01:45:36.2557119Z --> Processing Dependency: cpp = 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-11-23T01:45:36.2576149Z --> Processing Dependency: libsanitizer >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-11-23T01:45:36.2630777Z --> Processing Dependency: libquadmath >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-11-23T01:45:36.2683728Z --> Processing Dependency: libmpx >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-11-23T01:45:36.2736944Z --> Processing Dependency: libitm >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-11-23T01:45:36.2788748Z --> Processing Dependency: libcilkrts >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-11-23T01:45:36.2841065Z --> Processing Dependency: libatomic >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-11-23T01:45:36.2894759Z --> Processing Dependency: glibc-devel >= 2.2.90-12 for package: gcc-7.3.1-15.amzn2.x86_64
2022-11-23T01:45:36.3057178Z --> Processing Dependency: libmpfr.so.4()(64bit) for package: gcc-7.3.1-15.amzn2.x86_64
2022-11-23T01:45:36.3078613Z --> Processing Dependency: libmpc.so.3()(64bit) for package: gcc-7.3.1-15.amzn2.x86_64
2022-11-23T01:45:36.3100020Z ---> Package gcc-c++.x86_64 0:7.3.1-15.amzn2 will be installed
2022-11-23T01:45:36.3126641Z ---> Package gcc-gfortran.x86_64 0:7.3.1-15.amzn2 will be installed
2022-11-23T01:45:36.3160856Z --> Processing Dependency: libgfortran.so.4()(64bit) for package: gcc-gfortran-7.3.1-15.amzn2.x86_64
2022-11-23T01:45:36.3223120Z ---> Package indent.x86_64 0:2.2.11-13.amzn2.0.2 will be installed
2022-11-23T01:45:36.3237135Z ---> Package intltool.noarch 0:0.50.2-7.amzn2 will be installed
2022-11-23T01:45:36.3288373Z --> Processing Dependency: perl(XML::Parser) for package: intltool-0.50.2-7.amzn2.noarch
2022-11-23T01:45:36.3304169Z --> Processing Dependency: gettext-devel for package: intltool-0.50.2-7.amzn2.noarch
2022-11-23T01:45:36.3322211Z ---> Package libtool.x86_64 0:2.4.2-22.2.amzn2.0.2 will be installed
2022-11-23T01:45:36.3351253Z ---> Package patch.x86_64 0:2.7.1-12.amzn2.0.2 will be installed
2022-11-23T01:45:36.3387669Z ---> Package patchutils.x86_64 0:0.3.3-4.amzn2.0.1 will be installed
2022-11-23T01:45:36.3412802Z ---> Package rcs.x86_64 0:5.9.0-5.amzn2.0.2 will be installed
2022-11-23T01:45:36.3444268Z ---> Package rpm-build.x86_64 0:4.11.3-48.amzn2.0.2 will be installed
2022-11-23T01:45:36.3684454Z --> Processing Dependency: /usr/bin/gdb-add-index for package: rpm-build-4.11.3-48.amzn2.0.2.x86_64
2022-11-23T01:45:36.3702715Z ---> Package rpm-sign.x86_64 0:4.11.3-48.amzn2.0.2 will be installed
2022-11-23T01:45:36.3725538Z ---> Package subversion.x86_64 0:1.7.14-16.amzn2.0.1 will be installed
2022-11-23T01:45:36.3896788Z --> Processing Dependency: subversion-libs(x86-64) = 1.7.14-16.amzn2.0.1 for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:45:36.3915831Z --> Processing Dependency: libsvn_wc-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:45:36.3916508Z --> Processing Dependency: libsvn_subr-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:45:36.3917152Z --> Processing Dependency: libsvn_repos-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:45:36.3917792Z --> Processing Dependency: libsvn_ra_svn-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:45:36.3918405Z --> Processing Dependency: libsvn_ra_neon-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:45:36.3919039Z --> Processing Dependency: libsvn_ra_local-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:45:36.3919655Z --> Processing Dependency: libsvn_ra-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:45:36.3920281Z --> Processing Dependency: libsvn_fs_util-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:45:36.3920884Z --> Processing Dependency: libsvn_fs_fs-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:45:36.3921514Z --> Processing Dependency: libsvn_fs_base-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:45:36.3922119Z --> Processing Dependency: libsvn_fs-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:45:36.3922733Z --> Processing Dependency: libsvn_diff-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:45:36.3923333Z --> Processing Dependency: libsvn_delta-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:45:36.3923956Z --> Processing Dependency: libsvn_client-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:45:36.3924561Z --> Processing Dependency: libneon.so.27()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:45:36.3943634Z --> Processing Dependency: libaprutil-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:45:36.3963508Z --> Processing Dependency: libapr-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:45:36.3987566Z ---> Package swig.x86_64 0:3.0.12-11.amzn2.0.3 will be installed
2022-11-23T01:45:36.4008376Z ---> Package system-rpm-config.noarch 0:9.1.0-76.amzn2.0.14 will be installed
2022-11-23T01:45:36.4055149Z --> Processing Dependency: dwz >= 0.4 for package: system-rpm-config-9.1.0-76.amzn2.0.14.noarch
2022-11-23T01:45:36.4071878Z --> Processing Dependency: perl-srpm-macros for package: system-rpm-config-9.1.0-76.amzn2.0.14.noarch
2022-11-23T01:45:36.4083987Z --> Processing Dependency: go-srpm-macros for package: system-rpm-config-9.1.0-76.amzn2.0.14.noarch
2022-11-23T01:45:36.4259713Z ---> Package systemtap.x86_64 0:4.5-1.amzn2.0.1 will be installed
2022-11-23T01:45:36.4272376Z --> Processing Dependency: systemtap-devel = 4.5-1.amzn2.0.1 for package: systemtap-4.5-1.amzn2.0.1.x86_64
2022-11-23T01:45:36.4287068Z --> Processing Dependency: systemtap-client = 4.5-1.amzn2.0.1 for package: systemtap-4.5-1.amzn2.0.1.x86_64
2022-11-23T01:45:36.4301346Z --> Running transaction check
2022-11-23T01:45:36.4302861Z ---> Package apr.x86_64 0:1.7.0-9.amzn2 will be installed
2022-11-23T01:45:36.4385511Z ---> Package apr-util.x86_64 0:1.6.1-5.amzn2.0.2 will be installed
2022-11-23T01:45:36.4424567Z --> Processing Dependency: apr-util-bdb(x86-64) = 1.6.1-5.amzn2.0.2 for package: apr-util-1.6.1-5.amzn2.0.2.x86_64
2022-11-23T01:45:36.4438137Z ---> Package cpp.x86_64 0:7.3.1-15.amzn2 will be installed
2022-11-23T01:45:36.4514487Z ---> Package dwz.x86_64 0:0.11-3.amzn2.0.3 will be installed
2022-11-23T01:45:36.4525169Z ---> Package emacs-filesystem.noarch 1:27.2-4.amzn2.0.1 will be installed
2022-11-23T01:45:36.4525665Z ---> Package gdb.x86_64 0:8.0.1-36.amzn2.0.1 will be installed
2022-11-23T01:45:36.4595960Z ---> Package gettext-devel.x86_64 0:0.19.8.1-3.amzn2 will be installed
2022-11-23T01:45:36.4653933Z --> Processing Dependency: gettext-common-devel = 0.19.8.1-3.amzn2 for package: gettext-devel-0.19.8.1-3.amzn2.x86_64
2022-11-23T01:45:36.4663205Z ---> Package glibc-devel.x86_64 0:2.26-62.amzn2 will be installed
2022-11-23T01:45:36.4784328Z --> Processing Dependency: glibc-headers = 2.26-62.amzn2 for package: glibc-devel-2.26-62.amzn2.x86_64
2022-11-23T01:45:36.4812333Z --> Processing Dependency: glibc-headers for package: glibc-devel-2.26-62.amzn2.x86_64
2022-11-23T01:45:36.4812923Z ---> Package go-srpm-macros.noarch 0:3.0.15-23.amzn2.0.2 will be installed
2022-11-23T01:45:36.4818093Z ---> Package libatomic.x86_64 0:7.3.1-15.amzn2 will be installed
2022-11-23T01:45:36.4831505Z ---> Package libcilkrts.x86_64 0:7.3.1-15.amzn2 will be installed
2022-11-23T01:45:36.4858240Z ---> Package libgfortran.x86_64 0:7.3.1-15.amzn2 will be installed
2022-11-23T01:45:36.4893403Z ---> Package libitm.x86_64 0:7.3.1-15.amzn2 will be installed
2022-11-23T01:45:36.4909249Z ---> Package libmpc.x86_64 0:1.0.1-3.amzn2.0.2 will be installed
2022-11-23T01:45:36.4921584Z ---> Package libmpx.x86_64 0:7.3.1-15.amzn2 will be installed
2022-11-23T01:45:36.4935743Z ---> Package libquadmath.x86_64 0:7.3.1-15.amzn2 will be installed
2022-11-23T01:45:36.4961330Z ---> Package libsanitizer.x86_64 0:7.3.1-15.amzn2 will be installed
2022-11-23T01:45:36.5007013Z ---> Package m4.x86_64 0:1.4.16-10.amzn2.0.2 will be installed
2022-11-23T01:45:36.5022428Z ---> Package mpfr.x86_64 0:3.1.1-4.amzn2.0.2 will be installed
2022-11-23T01:45:36.5042569Z ---> Package neon.x86_64 0:0.30.0-3.amzn2.0.2 will be installed
2022-11-23T01:45:36.5118156Z --> Processing Dependency: libgnutls.so.28(GNUTLS_2_12)(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-11-23T01:45:36.5155518Z --> Processing Dependency: libgnutls.so.28(GNUTLS_1_4)(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-11-23T01:45:36.5156141Z --> Processing Dependency: libproxy.so.1()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-11-23T01:45:36.5175579Z --> Processing Dependency: libpakchois.so.0()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-11-23T01:45:36.5192756Z --> Processing Dependency: libgnutls.so.28()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-11-23T01:45:36.5198833Z ---> Package perl-Data-Dumper.x86_64 0:2.145-3.amzn2.0.2 will be installed
2022-11-23T01:45:36.5245933Z ---> Package perl-Test-Harness.noarch 0:3.28-3.amzn2 will be installed
2022-11-23T01:45:36.5339121Z ---> Package perl-Thread-Queue.noarch 0:3.02-2.amzn2 will be installed
2022-11-23T01:45:36.5351423Z ---> Package perl-XML-Parser.x86_64 0:2.41-10.amzn2.0.2 will be installed
2022-11-23T01:45:36.5366962Z ---> Package perl-srpm-macros.noarch 0:1-8.amzn2.0.1 will be installed
2022-11-23T01:45:36.5367511Z ---> Package subversion-libs.x86_64 0:1.7.14-16.amzn2.0.1 will be installed
2022-11-23T01:45:36.5396464Z ---> Package systemtap-client.x86_64 0:4.5-1.amzn2.0.1 will be installed
2022-11-23T01:45:36.5601414Z --> Processing Dependency: mokutil for package: systemtap-client-4.5-1.amzn2.0.1.x86_64
2022-11-23T01:45:36.5614608Z --> Processing Dependency: libavahi-common.so.3()(64bit) for package: systemtap-client-4.5-1.amzn2.0.1.x86_64
2022-11-23T01:45:36.5640699Z --> Processing Dependency: libavahi-client.so.3()(64bit) for package: systemtap-client-4.5-1.amzn2.0.1.x86_64
2022-11-23T01:45:36.5641346Z ---> Package systemtap-devel.x86_64 0:4.5-1.amzn2.0.1 will be installed
2022-11-23T01:45:36.5756087Z --> Processing Dependency: kernel-devel-uname-r for package: systemtap-devel-4.5-1.amzn2.0.1.x86_64
2022-11-23T01:45:36.6790188Z --> Running transaction check
2022-11-23T01:45:36.6790671Z ---> Package apr-util-bdb.x86_64 0:1.6.1-5.amzn2.0.2 will be installed
2022-11-23T01:45:36.6800570Z ---> Package avahi-libs.x86_64 0:0.6.31-20.amzn2 will be installed
2022-11-23T01:45:36.6826701Z ---> Package gettext-common-devel.noarch 0:0.19.8.1-3.amzn2 will be installed
2022-11-23T01:45:36.6827236Z ---> Package glibc-headers.x86_64 0:2.26-62.amzn2 will be installed
2022-11-23T01:45:36.6900585Z --> Processing Dependency: kernel-headers >= 2.2.1 for package: glibc-headers-2.26-62.amzn2.x86_64
2022-11-23T01:45:36.8003805Z --> Processing Dependency: kernel-headers for package: glibc-headers-2.26-62.amzn2.x86_64
2022-11-23T01:45:36.8004373Z ---> Package gnutls.x86_64 0:3.3.29-9.amzn2.0.1 will be installed
2022-11-23T01:45:36.8068974Z --> Processing Dependency: trousers >= 0.3.11.2 for package: gnutls-3.3.29-9.amzn2.0.1.x86_64
2022-11-23T01:45:36.8095154Z ---> Package kernel-devel.x86_64 0:4.14.296-222.539.amzn2 will be installed
2022-11-23T01:45:36.8120879Z --> Processing Dependency: elfutils-libelf-devel for package: kernel-devel-4.14.296-222.539.amzn2.x86_64
2022-11-23T01:45:36.8141033Z ---> Package libproxy.x86_64 0:0.4.11-10.amzn2.0.3 will be installed
2022-11-23T01:45:36.8167852Z --> Processing Dependency: libmodman.so.1()(64bit) for package: libproxy-0.4.11-10.amzn2.0.3.x86_64
2022-11-23T01:45:36.8185975Z ---> Package mokutil.x86_64 1:0.3.0-10.amzn2.0.1 will be installed
2022-11-23T01:45:36.8233410Z --> Processing Dependency: libefivar.so.1(libefivar.so.0)(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64
2022-11-23T01:45:36.8252857Z --> Processing Dependency: libefivar.so.1(LIBEFIVAR_0.24)(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64
2022-11-23T01:45:36.8253508Z --> Processing Dependency: libefivar.so.1()(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64
2022-11-23T01:45:36.8254032Z ---> Package pakchois.x86_64 0:0.4-10.amzn2.0.2 will be installed
2022-11-23T01:45:36.8268147Z --> Running transaction check
2022-11-23T01:45:36.8268618Z ---> Package efivar-libs.x86_64 0:31-4.amzn2.0.4 will be installed
2022-11-23T01:45:36.8285418Z ---> Package elfutils-libelf-devel.x86_64 0:0.176-2.amzn2 will be installed
2022-11-23T01:45:36.8296793Z --> Processing Dependency: pkgconfig(zlib) for package: elfutils-libelf-devel-0.176-2.amzn2.x86_64
2022-11-23T01:45:36.8327128Z ---> Package kernel-headers.x86_64 0:4.14.296-222.539.amzn2 will be installed
2022-11-23T01:45:36.8327657Z ---> Package libmodman.x86_64 0:2.0.1-8.amzn2.0.2 will be installed
2022-11-23T01:45:36.8345487Z ---> Package trousers.x86_64 0:0.3.14-2.amzn2.0.2 will be installed
2022-11-23T01:45:36.8401456Z --> Running transaction check
2022-11-23T01:45:36.8401935Z ---> Package zlib-devel.x86_64 0:1.2.7-19.amzn2.0.2 will be installed
2022-11-23T01:45:37.1020953Z --> Finished Dependency Resolution
2022-11-23T01:45:37.1785164Z 
2022-11-23T01:45:37.1785885Z Dependencies Resolved
2022-11-23T01:45:37.1896763Z 
2022-11-23T01:45:37.1897134Z ================================================================================
2022-11-23T01:45:37.1897943Z  Package                 Arch     Version                    Repository    Size
2022-11-23T01:45:37.1898340Z ================================================================================
2022-11-23T01:45:37.1898639Z Installing for group install "Development Tools":
2022-11-23T01:45:37.1899302Z  autoconf                noarch   2.69-11.amzn2              amzn2-core   701 k
2022-11-23T01:45:37.1899844Z  automake                noarch   1.13.4-3.1.amzn2           amzn2-core   679 k
2022-11-23T01:45:37.1900318Z  bison                   x86_64   3.0.4-6.amzn2.0.2          amzn2-core   674 k
2022-11-23T01:45:37.1900929Z  byacc                   x86_64   1.9.20130304-3.amzn2.0.2   amzn2-core    66 k
2022-11-23T01:45:37.1901496Z  cscope                  x86_64   15.8-10.amzn2.0.2          amzn2-core   204 k
2022-11-23T01:45:37.1901945Z  ctags                   x86_64   5.8-13.amzn2.0.2           amzn2-core   157 k
2022-11-23T01:45:37.1902353Z  diffstat                x86_64   1.57-4.amzn2.0.2           amzn2-core    35 k
2022-11-23T01:45:37.1902798Z  doxygen                 x86_64   1:1.8.5-4.amzn2            amzn2-core   3.5 M
2022-11-23T01:45:37.1903244Z  elfutils                x86_64   0.176-2.amzn2              amzn2-core   307 k
2022-11-23T01:45:37.1903674Z  flex                    x86_64   2.5.37-3.amzn2.0.3         amzn2-core   291 k
2022-11-23T01:45:37.1904165Z  gcc                     x86_64   7.3.1-15.amzn2             amzn2-core    22 M
2022-11-23T01:45:37.1904557Z  gcc-c++                 x86_64   7.3.1-15.amzn2             amzn2-core    13 M
2022-11-23T01:45:37.1904916Z  gcc-gfortran            x86_64   7.3.1-15.amzn2             amzn2-core    11 M
2022-11-23T01:45:37.1905338Z  indent                  x86_64   2.2.11-13.amzn2.0.2        amzn2-core   150 k
2022-11-23T01:45:37.1905763Z  intltool                noarch   0.50.2-7.amzn2             amzn2-core    59 k
2022-11-23T01:45:37.1906198Z  libtool                 x86_64   2.4.2-22.2.amzn2.0.2       amzn2-core   588 k
2022-11-23T01:45:37.1906616Z  patch                   x86_64   2.7.1-12.amzn2.0.2         amzn2-core   110 k
2022-11-23T01:45:37.1907056Z  patchutils              x86_64   0.3.3-4.amzn2.0.1          amzn2-core   104 k
2022-11-23T01:45:37.1907470Z  rcs                     x86_64   5.9.0-5.amzn2.0.2          amzn2-core   231 k
2022-11-23T01:45:37.1907903Z  rpm-build               x86_64   4.11.3-48.amzn2.0.2        amzn2-core   150 k
2022-11-23T01:45:37.1908324Z  rpm-sign                x86_64   4.11.3-48.amzn2.0.2        amzn2-core    50 k
2022-11-23T01:45:37.1908816Z  subversion              x86_64   1.7.14-16.amzn2.0.1        amzn2-core   1.0 M
2022-11-23T01:45:37.1909244Z  swig                    x86_64   3.0.12-11.amzn2.0.3        amzn2-core   1.4 M
2022-11-23T01:45:37.1909693Z  system-rpm-config       noarch   9.1.0-76.amzn2.0.14        amzn2-core    90 k
2022-11-23T01:45:37.1910133Z  systemtap               x86_64   4.5-1.amzn2.0.1            amzn2-core    12 k
2022-11-23T01:45:37.1910450Z Installing for dependencies:
2022-11-23T01:45:37.1910857Z  apr                     x86_64   1.7.0-9.amzn2              amzn2-core   122 k
2022-11-23T01:45:37.1911272Z  apr-util                x86_64   1.6.1-5.amzn2.0.2          amzn2-core    99 k
2022-11-23T01:45:37.1911723Z  apr-util-bdb            x86_64   1.6.1-5.amzn2.0.2          amzn2-core    19 k
2022-11-23T01:45:37.1912169Z  avahi-libs              x86_64   0.6.31-20.amzn2            amzn2-core    61 k
2022-11-23T01:45:37.1912603Z  cpp                     x86_64   7.3.1-15.amzn2             amzn2-core   9.2 M
2022-11-23T01:45:37.1913011Z  dwz                     x86_64   0.11-3.amzn2.0.3           amzn2-core    98 k
2022-11-23T01:45:37.1913482Z  efivar-libs             x86_64   31-4.amzn2.0.4             amzn2-core    68 k
2022-11-23T01:45:37.1913899Z  elfutils-libelf-devel   x86_64   0.176-2.amzn2              amzn2-core    40 k
2022-11-23T01:45:37.1914350Z  emacs-filesystem        noarch   1:27.2-4.amzn2.0.1         amzn2-core    67 k
2022-11-23T01:45:37.1914793Z  gdb                     x86_64   8.0.1-36.amzn2.0.1         amzn2-core   3.1 M
2022-11-23T01:45:37.1915245Z  gettext-common-devel    noarch   0.19.8.1-3.amzn2           amzn2-core   410 k
2022-11-23T01:45:37.1915709Z  gettext-devel           x86_64   0.19.8.1-3.amzn2           amzn2-core   320 k
2022-11-23T01:45:37.1916138Z  glibc-devel             x86_64   2.26-62.amzn2              amzn2-core   995 k
2022-11-23T01:45:37.1916656Z  glibc-headers           x86_64   2.26-62.amzn2              amzn2-core   516 k
2022-11-23T01:45:37.1917143Z  gnutls                  x86_64   3.3.29-9.amzn2.0.1         amzn2-core   661 k
2022-11-23T01:45:37.1917634Z  go-srpm-macros          noarch   3.0.15-23.amzn2.0.2        amzn2-core    23 k
2022-11-23T01:45:37.1918014Z  kernel-devel            x86_64   4.14.296-222.539.amzn2     amzn2-core    13 M
2022-11-23T01:45:37.1918460Z  kernel-headers          x86_64   4.14.296-222.539.amzn2     amzn2-core   1.2 M
2022-11-23T01:45:37.1918902Z  libatomic               x86_64   7.3.1-15.amzn2             amzn2-core    46 k
2022-11-23T01:45:37.1919315Z  libcilkrts              x86_64   7.3.1-15.amzn2             amzn2-core    85 k
2022-11-23T01:45:37.1919744Z  libgfortran             x86_64   7.3.1-15.amzn2             amzn2-core   536 k
2022-11-23T01:45:37.1920178Z  libitm                  x86_64   7.3.1-15.amzn2             amzn2-core    85 k
2022-11-23T01:45:37.1920655Z  libmodman               x86_64   2.0.1-8.amzn2.0.2          amzn2-core    29 k
2022-11-23T01:45:37.1921092Z  libmpc                  x86_64   1.0.1-3.amzn2.0.2          amzn2-core    52 k
2022-11-23T01:45:37.1921524Z  libmpx                  x86_64   7.3.1-15.amzn2             amzn2-core    51 k
2022-11-23T01:45:37.1921955Z  libproxy                x86_64   0.4.11-10.amzn2.0.3        amzn2-core    61 k
2022-11-23T01:45:37.1922382Z  libquadmath             x86_64   7.3.1-15.amzn2             amzn2-core   189 k
2022-11-23T01:45:37.1922821Z  libsanitizer            x86_64   7.3.1-15.amzn2             amzn2-core   642 k
2022-11-23T01:45:37.1923253Z  m4                      x86_64   1.4.16-10.amzn2.0.2        amzn2-core   256 k
2022-11-23T01:45:37.1923659Z  mokutil                 x86_64   1:0.3.0-10.amzn2.0.1       amzn2-core    39 k
2022-11-23T01:45:37.1924081Z  mpfr                    x86_64   3.1.1-4.amzn2.0.2          amzn2-core   208 k
2022-11-23T01:45:37.1924504Z  neon                    x86_64   0.30.0-3.amzn2.0.2         amzn2-core   166 k
2022-11-23T01:45:37.1924935Z  pakchois                x86_64   0.4-10.amzn2.0.2           amzn2-core    14 k
2022-11-23T01:45:37.1925375Z  perl-Data-Dumper        x86_64   2.145-3.amzn2.0.2          amzn2-core    48 k
2022-11-23T01:45:37.1925845Z  perl-Test-Harness       noarch   3.28-3.amzn2               amzn2-core   302 k
2022-11-23T01:45:37.1926315Z  perl-Thread-Queue       noarch   3.02-2.amzn2               amzn2-core    17 k
2022-11-23T01:45:37.1926774Z  perl-XML-Parser         x86_64   2.41-10.amzn2.0.2          amzn2-core   223 k
2022-11-23T01:45:37.1927322Z  perl-srpm-macros        noarch   1-8.amzn2.0.1              amzn2-core   4.7 k
2022-11-23T01:45:37.1927822Z  subversion-libs         x86_64   1.7.14-16.amzn2.0.1        amzn2-core   912 k
2022-11-23T01:45:37.1928278Z  systemtap-client        x86_64   4.5-1.amzn2.0.1            amzn2-core   3.7 M
2022-11-23T01:45:37.1928719Z  systemtap-devel         x86_64   4.5-1.amzn2.0.1            amzn2-core   2.3 M
2022-11-23T01:45:37.1929278Z  trousers                x86_64   0.3.14-2.amzn2.0.2         amzn2-core   294 k
2022-11-23T01:45:37.1929598Z  zlib-devel              x86_64   1.2.7-19.amzn2.0.2         amzn2-core    50 k
2022-11-23T01:45:37.1929808Z 
2022-11-23T01:45:37.1929908Z Transaction Summary
2022-11-23T01:45:37.1930273Z ================================================================================
2022-11-23T01:45:37.1930589Z Install  25 Packages (+43 Dependent packages)
2022-11-23T01:45:37.1930789Z 
2022-11-23T01:45:37.1930918Z Total download size: 96 M
2022-11-23T01:45:37.1931166Z Installed size: 303 M
2022-11-23T01:45:37.1931429Z Downloading packages:
2022-11-23T01:45:37.1947281Z Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
2022-11-23T01:45:38.8983172Z --------------------------------------------------------------------------------
2022-11-23T01:45:38.8983641Z Total                                               56 MB/s |  96 MB  00:01     
2022-11-23T01:45:39.0079383Z Running transaction check
2022-11-23T01:45:39.0871730Z Running transaction test
2022-11-23T01:45:39.5110152Z Transaction test succeeded
2022-11-23T01:45:39.5113148Z Running transaction
2022-11-23T01:45:44.8249439Z   Installing : mpfr-3.1.1-4.amzn2.0.2.x86_64                               1/68 
2022-11-23T01:45:47.5013913Z   Installing : libmpc-1.0.1-3.amzn2.0.2.x86_64                             2/68 
2022-11-23T01:45:49.8112567Z   Installing : m4-1.4.16-10.amzn2.0.2.x86_64                               3/68 
2022-11-23T01:45:52.2744290Z   Installing : apr-1.7.0-9.amzn2.x86_64                                    4/68 
2022-11-23T01:45:54.6962257Z   Installing : apr-util-bdb-1.6.1-5.amzn2.0.2.x86_64                       5/68 
2022-11-23T01:45:57.1653366Z   Installing : apr-util-1.6.1-5.amzn2.0.2.x86_64                           6/68 
2022-11-23T01:45:59.6803560Z   Installing : avahi-libs-0.6.31-20.amzn2.x86_64                           7/68 
2022-11-23T01:46:01.1940864Z   Installing : libquadmath-7.3.1-15.amzn2.x86_64                           8/68 
2022-11-23T01:46:01.2198316Z   Installing : patch-2.7.1-12.amzn2.0.2.x86_64                             9/68 
2022-11-23T01:46:01.3094088Z   Installing : perl-Thread-Queue-3.02-2.amzn2.noarch                      10/68 
2022-11-23T01:46:02.3994888Z   Installing : libgfortran-7.3.1-15.amzn2.x86_64                          11/68 
2022-11-23T01:46:02.4421469Z   Installing : cpp-7.3.1-15.amzn2.x86_64                                  12/68 
2022-11-23T01:46:02.4840738Z   Installing : libmodman-2.0.1-8.amzn2.0.2.x86_64                         13/68 
2022-11-23T01:46:02.5490346Z   Installing : libproxy-0.4.11-10.amzn2.0.3.x86_64                        14/68 
2022-11-23T01:46:02.6107703Z   Installing : perl-XML-Parser-2.41-10.amzn2.0.2.x86_64                   15/68 
2022-11-23T01:46:02.7234101Z   Installing : elfutils-0.176-2.amzn2.x86_64                              16/68 
2022-11-23T01:46:02.7595622Z   Installing : libsanitizer-7.3.1-15.amzn2.x86_64                         17/68 
2022-11-23T01:46:02.7879667Z   Installing : 1:emacs-filesystem-27.2-4.amzn2.0.1.noarch                 18/68 
2022-11-23T01:46:02.8242383Z   Installing : efivar-libs-31-4.amzn2.0.4.x86_64                          19/68 
2022-11-23T01:46:02.8567946Z   Installing : 1:mokutil-0.3.0-10.amzn2.0.1.x86_64                        20/68 
2022-11-23T01:46:02.9455947Z   Installing : gettext-common-devel-0.19.8.1-3.amzn2.noarch               21/68 
2022-11-23T01:46:03.0154250Z   Installing : gettext-devel-0.19.8.1-3.amzn2.x86_64                      22/68 
2022-11-23T01:46:03.1679749Z   Installing : dwz-0.11-3.amzn2.0.3.x86_64                                23/68 
2022-11-23T01:46:03.3271678Z   Installing : trousers-0.3.14-2.amzn2.0.2.x86_64                         24/68 
2022-11-23T01:46:03.3750961Z   Installing : gnutls-3.3.29-9.amzn2.0.1.x86_64                           25/68 
2022-11-23T01:46:03.7887560Z   Installing : libitm-7.3.1-15.amzn2.x86_64                               26/68 
2022-11-23T01:46:03.8221270Z   Installing : gdb-8.0.1-36.amzn2.0.1.x86_64                              27/68 
2022-11-23T01:46:03.8549614Z   Installing : libmpx-7.3.1-15.amzn2.x86_64                               28/68 
2022-11-23T01:46:03.8805729Z   Installing : perl-srpm-macros-1-8.amzn2.0.1.noarch                      29/68 
2022-11-23T01:46:03.9180079Z   Installing : go-srpm-macros-3.0.15-23.amzn2.0.2.noarch                  30/68 
2022-11-23T01:46:03.9476392Z   Installing : system-rpm-config-9.1.0-76.amzn2.0.14.noarch               31/68 
2022-11-23T01:46:04.0442876Z   Installing : perl-Data-Dumper-2.145-3.amzn2.0.2.x86_64                  32/68 
2022-11-23T01:46:04.1375550Z   Installing : autoconf-2.69-11.amzn2.noarch                              33/68 
2022-11-23T01:46:04.2493739Z   Installing : perl-Test-Harness-3.28-3.amzn2.noarch                      34/68 
2022-11-23T01:46:04.2964473Z   Installing : automake-1.13.4-3.1.amzn2.noarch                           35/68 
2022-11-23T01:46:04.3258620Z   Installing : zlib-devel-1.2.7-19.amzn2.0.2.x86_64                       36/68 
2022-11-23T01:46:04.3542845Z   Installing : elfutils-libelf-devel-0.176-2.amzn2.x86_64                 37/68 
2022-11-23T01:46:04.6655467Z   Installing : libatomic-7.3.1-15.amzn2.x86_64                            38/68 
2022-11-23T01:46:04.8502337Z   Installing : kernel-headers-4.14.296-222.539.amzn2.x86_64               39/68 
2022-11-23T01:46:04.9932421Z   Installing : glibc-headers-2.26-62.amzn2.x86_64                         40/68 
2022-11-23T01:46:05.0386438Z   Installing : glibc-devel-2.26-62.amzn2.x86_64                           41/68 
2022-11-23T01:46:07.1199807Z   Installing : libcilkrts-7.3.1-15.amzn2.x86_64                           42/68 
2022-11-23T01:46:11.1168844Z   Installing : gcc-7.3.1-15.amzn2.x86_64                                  43/68 
2022-11-23T01:46:25.0441116Z   Installing : kernel-devel-4.14.296-222.539.amzn2.x86_64                 44/68 
2022-11-23T01:46:25.6801585Z   Installing : systemtap-devel-4.5-1.amzn2.0.1.x86_64                     45/68 
2022-11-23T01:46:25.7356421Z   Installing : systemtap-client-4.5-1.amzn2.0.1.x86_64                    46/68 
2022-11-23T01:46:25.7988574Z   Installing : pakchois-0.4-10.amzn2.0.2.x86_64                           47/68 
2022-11-23T01:46:25.9414908Z   Installing : neon-0.30.0-3.amzn2.0.2.x86_64                             48/68 
2022-11-23T01:46:26.1258961Z   Installing : subversion-libs-1.7.14-16.amzn2.0.1.x86_64                 49/68 
2022-11-23T01:46:26.3186303Z   Installing : subversion-1.7.14-16.amzn2.0.1.x86_64                      50/68 
2022-11-23T01:46:27.5725182Z   Installing : systemtap-4.5-1.amzn2.0.1.x86_64                           51/68 
2022-11-23T01:46:29.2427289Z   Installing : gcc-gfortran-7.3.1-15.amzn2.x86_64                         52/68 
2022-11-23T01:46:29.3678797Z   Installing : gcc-c++-7.3.1-15.amzn2.x86_64                              53/68 
2022-11-23T01:46:29.4159067Z   Installing : libtool-2.4.2-22.2.amzn2.0.2.x86_64                        54/68 
2022-11-23T01:46:29.4574263Z   Installing : intltool-0.50.2-7.amzn2.noarch                             55/68 
2022-11-23T01:46:29.5171501Z   Installing : rpm-build-4.11.3-48.amzn2.0.2.x86_64                       56/68 
2022-11-23T01:46:29.5835153Z   Installing : cscope-15.8-10.amzn2.0.2.x86_64                            57/68 
2022-11-23T01:46:29.7062291Z   Installing : flex-2.5.37-3.amzn2.0.3.x86_64                             58/68 
2022-11-23T01:46:29.7735902Z   Installing : bison-3.0.4-6.amzn2.0.2.x86_64                             59/68 
2022-11-23T01:46:29.8290071Z   Installing : rcs-5.9.0-5.amzn2.0.2.x86_64                               60/68 
2022-11-23T01:46:29.8690574Z   Installing : ctags-5.8-13.amzn2.0.2.x86_64                              61/68 
2022-11-23T01:46:29.9222320Z   Installing : indent-2.2.11-13.amzn2.0.2.x86_64                          62/68 
2022-11-23T01:46:30.6452151Z   Installing : patchutils-0.3.3-4.amzn2.0.1.x86_64                        63/68 
2022-11-23T01:46:30.6896790Z   Installing : 1:doxygen-1.8.5-4.amzn2.x86_64                             64/68 
2022-11-23T01:46:30.7205511Z   Installing : diffstat-1.57-4.amzn2.0.2.x86_64                           65/68 
2022-11-23T01:46:31.0480809Z   Installing : byacc-1.9.20130304-3.amzn2.0.2.x86_64                      66/68 
2022-11-23T01:46:31.0953378Z   Installing : swig-3.0.12-11.amzn2.0.3.x86_64                            67/68 
2022-11-23T01:46:31.1676661Z   Installing : rpm-sign-4.11.3-48.amzn2.0.2.x86_64                        68/68 
2022-11-23T01:46:31.1819974Z   Verifying  : elfutils-libelf-devel-0.176-2.amzn2.x86_64                  1/68 
2022-11-23T01:46:31.1967564Z   Verifying  : perl-Thread-Queue-3.02-2.amzn2.noarch                       2/68 
2022-11-23T01:46:31.2134232Z   Verifying  : gettext-devel-0.19.8.1-3.amzn2.x86_64                       3/68 
2022-11-23T01:46:31.2295580Z   Verifying  : patch-2.7.1-12.amzn2.0.2.x86_64                             4/68 
2022-11-23T01:46:31.2652954Z   Verifying  : kernel-devel-4.14.296-222.539.amzn2.x86_64                  5/68 
2022-11-23T01:46:31.3052387Z   Verifying  : flex-2.5.37-3.amzn2.0.3.x86_64                              6/68 
2022-11-23T01:46:31.3178501Z   Verifying  : pakchois-0.4-10.amzn2.0.2.x86_64                            7/68 
2022-11-23T01:46:31.3316143Z   Verifying  : rpm-sign-4.11.3-48.amzn2.0.2.x86_64                         8/68 
2022-11-23T01:46:31.3424892Z   Verifying  : glibc-devel-2.26-62.amzn2.x86_64                            9/68 
2022-11-23T01:46:31.3547461Z   Verifying  : gcc-gfortran-7.3.1-15.amzn2.x86_64                         10/68 
2022-11-23T01:46:31.3658705Z   Verifying  : swig-3.0.12-11.amzn2.0.3.x86_64                            11/68 
2022-11-23T01:46:31.3777273Z   Verifying  : byacc-1.9.20130304-3.amzn2.0.2.x86_64                      12/68 
2022-11-23T01:46:31.3896056Z   Verifying  : libmpc-1.0.1-3.amzn2.0.2.x86_64                            13/68 
2022-11-23T01:46:31.4011811Z   Verifying  : libcilkrts-7.3.1-15.amzn2.x86_64                           14/68 
2022-11-23T01:46:31.4126805Z   Verifying  : kernel-headers-4.14.296-222.539.amzn2.x86_64               15/68 
2022-11-23T01:46:31.4227439Z   Verifying  : libproxy-0.4.11-10.amzn2.0.3.x86_64                        16/68 
2022-11-23T01:46:31.4350530Z   Verifying  : cscope-15.8-10.amzn2.0.2.x86_64                            17/68 
2022-11-23T01:46:31.4466274Z   Verifying  : diffstat-1.57-4.amzn2.0.2.x86_64                           18/68 
2022-11-23T01:46:31.4587418Z   Verifying  : 1:doxygen-1.8.5-4.amzn2.x86_64                             19/68 
2022-11-23T01:46:31.4705568Z   Verifying  : gcc-c++-7.3.1-15.amzn2.x86_64                              20/68 
2022-11-23T01:46:31.4821087Z   Verifying  : libatomic-7.3.1-15.amzn2.x86_64                            21/68 
2022-11-23T01:46:31.4946748Z   Verifying  : system-rpm-config-9.1.0-76.amzn2.0.14.noarch               22/68 
2022-11-23T01:46:31.5058925Z   Verifying  : systemtap-devel-4.5-1.amzn2.0.1.x86_64                     23/68 
2022-11-23T01:46:31.5179039Z   Verifying  : zlib-devel-1.2.7-19.amzn2.0.2.x86_64                       24/68 
2022-11-23T01:46:31.5309450Z   Verifying  : glibc-headers-2.26-62.amzn2.x86_64                         25/68 
2022-11-23T01:46:31.5413704Z   Verifying  : perl-Test-Harness-3.28-3.amzn2.noarch                      26/68 
2022-11-23T01:46:31.5559251Z   Verifying  : autoconf-2.69-11.amzn2.noarch                              27/68 
2022-11-23T01:46:31.5698446Z   Verifying  : libquadmath-7.3.1-15.amzn2.x86_64                          28/68 
2022-11-23T01:46:31.5831886Z   Verifying  : intltool-0.50.2-7.amzn2.noarch                             29/68 
2022-11-23T01:46:31.5941307Z   Verifying  : apr-util-1.6.1-5.amzn2.0.2.x86_64                          30/68 
2022-11-23T01:46:31.6066427Z   Verifying  : cpp-7.3.1-15.amzn2.x86_64                                  31/68 
2022-11-23T01:46:31.6186445Z   Verifying  : rpm-build-4.11.3-48.amzn2.0.2.x86_64                       32/68 
2022-11-23T01:46:31.6307192Z   Verifying  : go-srpm-macros-3.0.15-23.amzn2.0.2.noarch                  33/68 
2022-11-23T01:46:31.6427782Z   Verifying  : perl-Data-Dumper-2.145-3.amzn2.0.2.x86_64                  34/68 
2022-11-23T01:46:31.6545170Z   Verifying  : perl-srpm-macros-1-8.amzn2.0.1.noarch                      35/68 
2022-11-23T01:46:31.6679851Z   Verifying  : gnutls-3.3.29-9.amzn2.0.1.x86_64                           36/68 
2022-11-23T01:46:31.6820621Z   Verifying  : subversion-libs-1.7.14-16.amzn2.0.1.x86_64                 37/68 
2022-11-23T01:46:31.6966987Z   Verifying  : automake-1.13.4-3.1.amzn2.noarch                           38/68 
2022-11-23T01:46:31.7110742Z   Verifying  : apr-util-bdb-1.6.1-5.amzn2.0.2.x86_64                      39/68 
2022-11-23T01:46:31.7228604Z   Verifying  : libmpx-7.3.1-15.amzn2.x86_64                               40/68 
2022-11-23T01:46:31.7374157Z   Verifying  : avahi-libs-0.6.31-20.amzn2.x86_64                          41/68 
2022-11-23T01:46:31.7507911Z   Verifying  : bison-3.0.4-6.amzn2.0.2.x86_64                             42/68 
2022-11-23T01:46:31.7631590Z   Verifying  : libgfortran-7.3.1-15.amzn2.x86_64                          43/68 
2022-11-23T01:46:31.7779629Z   Verifying  : gdb-8.0.1-36.amzn2.0.1.x86_64                              44/68 
2022-11-23T01:46:31.7906364Z   Verifying  : patchutils-0.3.3-4.amzn2.0.1.x86_64                        45/68 
2022-11-23T01:46:31.8022249Z   Verifying  : libitm-7.3.1-15.amzn2.x86_64                               46/68 
2022-11-23T01:46:31.8145176Z   Verifying  : libtool-2.4.2-22.2.amzn2.0.2.x86_64                        47/68 
2022-11-23T01:46:31.8258613Z   Verifying  : gcc-7.3.1-15.amzn2.x86_64                                  48/68 
2022-11-23T01:46:31.8378377Z   Verifying  : indent-2.2.11-13.amzn2.0.2.x86_64                          49/68 
2022-11-23T01:46:31.8496803Z   Verifying  : subversion-1.7.14-16.amzn2.0.1.x86_64                      50/68 
2022-11-23T01:46:31.8610272Z   Verifying  : apr-1.7.0-9.amzn2.x86_64                                   51/68 
2022-11-23T01:46:31.8743762Z   Verifying  : ctags-5.8-13.amzn2.0.2.x86_64                              52/68 
2022-11-23T01:46:31.8877663Z   Verifying  : 1:mokutil-0.3.0-10.amzn2.0.1.x86_64                        53/68 
2022-11-23T01:46:31.8980895Z   Verifying  : mpfr-3.1.1-4.amzn2.0.2.x86_64                              54/68 
2022-11-23T01:46:31.9086260Z   Verifying  : trousers-0.3.14-2.amzn2.0.2.x86_64                         55/68 
2022-11-23T01:46:31.9206863Z   Verifying  : neon-0.30.0-3.amzn2.0.2.x86_64                             56/68 
2022-11-23T01:46:31.9330849Z   Verifying  : systemtap-4.5-1.amzn2.0.1.x86_64                           57/68 
2022-11-23T01:46:31.9446840Z   Verifying  : dwz-0.11-3.amzn2.0.3.x86_64                                58/68 
2022-11-23T01:46:31.9566010Z   Verifying  : gettext-common-devel-0.19.8.1-3.amzn2.noarch               59/68 
2022-11-23T01:46:31.9694998Z   Verifying  : systemtap-client-4.5-1.amzn2.0.1.x86_64                    60/68 
2022-11-23T01:46:31.9814605Z   Verifying  : efivar-libs-31-4.amzn2.0.4.x86_64                          61/68 
2022-11-23T01:46:31.9934852Z   Verifying  : rcs-5.9.0-5.amzn2.0.2.x86_64                               62/68 
2022-11-23T01:46:32.0070975Z   Verifying  : 1:emacs-filesystem-27.2-4.amzn2.0.1.noarch                 63/68 
2022-11-23T01:46:32.0200555Z   Verifying  : libsanitizer-7.3.1-15.amzn2.x86_64                         64/68 
2022-11-23T01:46:32.0316106Z   Verifying  : elfutils-0.176-2.amzn2.x86_64                              65/68 
2022-11-23T01:46:32.0427701Z   Verifying  : m4-1.4.16-10.amzn2.0.2.x86_64                              66/68 
2022-11-23T01:46:32.0537353Z   Verifying  : perl-XML-Parser-2.41-10.amzn2.0.2.x86_64                   67/68 
2022-11-23T01:46:32.1565950Z   Verifying  : libmodman-2.0.1-8.amzn2.0.2.x86_64                         68/68 
2022-11-23T01:46:32.1566443Z 
2022-11-23T01:46:32.1569352Z Installed:
2022-11-23T01:46:32.1570205Z   autoconf.noarch 0:2.69-11.amzn2                                               
2022-11-23T01:46:32.1570688Z   automake.noarch 0:1.13.4-3.1.amzn2                                            
2022-11-23T01:46:32.1571133Z   bison.x86_64 0:3.0.4-6.amzn2.0.2                                              
2022-11-23T01:46:32.1572346Z   byacc.x86_64 0:1.9.20130304-3.amzn2.0.2                                       
2022-11-23T01:46:32.1572844Z   cscope.x86_64 0:15.8-10.amzn2.0.2                                             
2022-11-23T01:46:32.1573288Z   ctags.x86_64 0:5.8-13.amzn2.0.2                                               
2022-11-23T01:46:32.1573718Z   diffstat.x86_64 0:1.57-4.amzn2.0.2                                            
2022-11-23T01:46:32.1574159Z   doxygen.x86_64 1:1.8.5-4.amzn2                                                
2022-11-23T01:46:32.1574695Z   elfutils.x86_64 0:0.176-2.amzn2                                               
2022-11-23T01:46:32.1575026Z   flex.x86_64 0:2.5.37-3.amzn2.0.3                                              
2022-11-23T01:46:32.1575433Z   gcc.x86_64 0:7.3.1-15.amzn2                                                   
2022-11-23T01:46:32.1575872Z   gcc-c++.x86_64 0:7.3.1-15.amzn2                                               
2022-11-23T01:46:32.1576311Z   gcc-gfortran.x86_64 0:7.3.1-15.amzn2                                          
2022-11-23T01:46:32.1577363Z   indent.x86_64 0:2.2.11-13.amzn2.0.2                                           
2022-11-23T01:46:32.1577867Z   intltool.noarch 0:0.50.2-7.amzn2                                              
2022-11-23T01:46:32.1578285Z   libtool.x86_64 0:2.4.2-22.2.amzn2.0.2                                         
2022-11-23T01:46:32.1578703Z   patch.x86_64 0:2.7.1-12.amzn2.0.2                                             
2022-11-23T01:46:32.1579163Z   patchutils.x86_64 0:0.3.3-4.amzn2.0.1                                         
2022-11-23T01:46:32.1579622Z   rcs.x86_64 0:5.9.0-5.amzn2.0.2                                                
2022-11-23T01:46:32.1580006Z   rpm-build.x86_64 0:4.11.3-48.amzn2.0.2                                        
2022-11-23T01:46:32.1580559Z   rpm-sign.x86_64 0:4.11.3-48.amzn2.0.2                                         
2022-11-23T01:46:32.1581003Z   subversion.x86_64 0:1.7.14-16.amzn2.0.1                                       
2022-11-23T01:46:32.1581446Z   swig.x86_64 0:3.0.12-11.amzn2.0.3                                             
2022-11-23T01:46:32.1581902Z   system-rpm-config.noarch 0:9.1.0-76.amzn2.0.14                                
2022-11-23T01:46:32.1582376Z   systemtap.x86_64 0:4.5-1.amzn2.0.1                                            
2022-11-23T01:46:32.1582559Z 
2022-11-23T01:46:32.1582695Z Dependency Installed:
2022-11-23T01:46:32.1583096Z   apr.x86_64 0:1.7.0-9.amzn2                                                    
2022-11-23T01:46:32.1583530Z   apr-util.x86_64 0:1.6.1-5.amzn2.0.2                                           
2022-11-23T01:46:32.1583957Z   apr-util-bdb.x86_64 0:1.6.1-5.amzn2.0.2                                       
2022-11-23T01:46:32.1584414Z   avahi-libs.x86_64 0:0.6.31-20.amzn2                                           
2022-11-23T01:46:32.1584940Z   cpp.x86_64 0:7.3.1-15.amzn2                                                   
2022-11-23T01:46:32.1585437Z   dwz.x86_64 0:0.11-3.amzn2.0.3                                                 
2022-11-23T01:46:32.1585768Z   efivar-libs.x86_64 0:31-4.amzn2.0.4                                           
2022-11-23T01:46:32.1586230Z   elfutils-libelf-devel.x86_64 0:0.176-2.amzn2                                  
2022-11-23T01:46:32.1586699Z   emacs-filesystem.noarch 1:27.2-4.amzn2.0.1                                    
2022-11-23T01:46:32.1587123Z   gdb.x86_64 0:8.0.1-36.amzn2.0.1                                               
2022-11-23T01:46:32.1587583Z   gettext-common-devel.noarch 0:0.19.8.1-3.amzn2                                
2022-11-23T01:46:32.1588177Z   gettext-devel.x86_64 0:0.19.8.1-3.amzn2                                       
2022-11-23T01:46:32.1588579Z   glibc-devel.x86_64 0:2.26-62.amzn2                                            
2022-11-23T01:46:32.1588920Z   glibc-headers.x86_64 0:2.26-62.amzn2                                          
2022-11-23T01:46:32.1589355Z   gnutls.x86_64 0:3.3.29-9.amzn2.0.1                                            
2022-11-23T01:46:32.1589834Z   go-srpm-macros.noarch 0:3.0.15-23.amzn2.0.2                                   
2022-11-23T01:46:32.1590283Z   kernel-devel.x86_64 0:4.14.296-222.539.amzn2                                  
2022-11-23T01:46:32.1590740Z   kernel-headers.x86_64 0:4.14.296-222.539.amzn2                                
2022-11-23T01:46:32.1591186Z   libatomic.x86_64 0:7.3.1-15.amzn2                                             
2022-11-23T01:46:32.1591608Z   libcilkrts.x86_64 0:7.3.1-15.amzn2                                            
2022-11-23T01:46:32.1592014Z   libgfortran.x86_64 0:7.3.1-15.amzn2                                           
2022-11-23T01:46:32.1592437Z   libitm.x86_64 0:7.3.1-15.amzn2                                                
2022-11-23T01:46:32.1592951Z   libmodman.x86_64 0:2.0.1-8.amzn2.0.2                                          
2022-11-23T01:46:32.1593267Z   libmpc.x86_64 0:1.0.1-3.amzn2.0.2                                             
2022-11-23T01:46:32.1593757Z   libmpx.x86_64 0:7.3.1-15.amzn2                                                
2022-11-23T01:46:32.1594197Z   libproxy.x86_64 0:0.4.11-10.amzn2.0.3                                         
2022-11-23T01:46:32.1594629Z   libquadmath.x86_64 0:7.3.1-15.amzn2                                           
2022-11-23T01:46:32.1595040Z   libsanitizer.x86_64 0:7.3.1-15.amzn2                                          
2022-11-23T01:46:32.1595459Z   m4.x86_64 0:1.4.16-10.amzn2.0.2                                               
2022-11-23T01:46:32.1595878Z   mokutil.x86_64 1:0.3.0-10.amzn2.0.1                                           
2022-11-23T01:46:32.1596361Z   mpfr.x86_64 0:3.1.1-4.amzn2.0.2                                               
2022-11-23T01:46:32.1596897Z   neon.x86_64 0:0.30.0-3.amzn2.0.2                                              
2022-11-23T01:46:32.1597214Z   pakchois.x86_64 0:0.4-10.amzn2.0.2                                            
2022-11-23T01:46:32.1597674Z   perl-Data-Dumper.x86_64 0:2.145-3.amzn2.0.2                                   
2022-11-23T01:46:32.1598231Z   perl-Test-Harness.noarch 0:3.28-3.amzn2                                       
2022-11-23T01:46:32.1598722Z   perl-Thread-Queue.noarch 0:3.02-2.amzn2                                       
2022-11-23T01:46:32.1599236Z   perl-XML-Parser.x86_64 0:2.41-10.amzn2.0.2                                    
2022-11-23T01:46:32.1599752Z   perl-srpm-macros.noarch 0:1-8.amzn2.0.1                                       
2022-11-23T01:46:32.1600228Z   subversion-libs.x86_64 0:1.7.14-16.amzn2.0.1                                  
2022-11-23T01:46:32.1600612Z   systemtap-client.x86_64 0:4.5-1.amzn2.0.1                                     
2022-11-23T01:46:32.1601036Z   systemtap-devel.x86_64 0:4.5-1.amzn2.0.1                                      
2022-11-23T01:46:32.1601457Z   trousers.x86_64 0:0.3.14-2.amzn2.0.2                                          
2022-11-23T01:46:32.1601940Z   zlib-devel.x86_64 0:1.2.7-19.amzn2.0.2                                        
2022-11-23T01:46:32.1602094Z 
2022-11-23T01:46:32.1602205Z Complete!
2022-11-23T01:46:32.2020590Z ++ uname -r
2022-11-23T01:46:32.2033983Z + sudo yum install -y 'kernel-devel-uname-r == 4.14.252-195.483.amzn2.x86_64'
2022-11-23T01:46:32.7319786Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-11-23T01:46:33.0288944Z Resolving Dependencies
2022-11-23T01:46:33.0295745Z --> Running transaction check
2022-11-23T01:46:33.0296387Z ---> Package kernel-devel.x86_64 0:4.14.252-195.483.amzn2 will be installed
2022-11-23T01:46:33.3237159Z --> Finished Dependency Resolution
2022-11-23T01:46:33.4096670Z 
2022-11-23T01:46:33.4098027Z Dependencies Resolved
2022-11-23T01:46:33.4102222Z 
2022-11-23T01:46:33.4102449Z ================================================================================
2022-11-23T01:46:33.4103302Z  Package          Arch       Version                       Repository      Size
2022-11-23T01:46:33.4104048Z ================================================================================
2022-11-23T01:46:33.4104615Z Installing:
2022-11-23T01:46:33.4105128Z  kernel-devel     x86_64     4.14.252-195.483.amzn2        amzn2-core      13 M
2022-11-23T01:46:33.4105356Z 
2022-11-23T01:46:33.4105477Z Transaction Summary
2022-11-23T01:46:33.4105800Z ================================================================================
2022-11-23T01:46:33.4106194Z Install  1 Package
2022-11-23T01:46:33.4106370Z 
2022-11-23T01:46:33.4106497Z Total download size: 13 M
2022-11-23T01:46:33.4106742Z Installed size: 60 M
2022-11-23T01:46:33.4106913Z Downloading packages:
2022-11-23T01:46:33.4117140Z Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
2022-11-23T01:46:33.7384627Z Running transaction check
2022-11-23T01:46:33.7576000Z Running transaction test
2022-11-23T01:46:34.1700312Z Transaction test succeeded
2022-11-23T01:46:34.1702118Z Running transaction
2022-11-23T01:46:52.3404081Z   Installing : kernel-devel-4.14.252-195.483.amzn2.x86_64                   1/1 
2022-11-23T01:46:52.4252007Z   Verifying  : kernel-devel-4.14.252-195.483.amzn2.x86_64                   1/1 
2022-11-23T01:46:52.4252343Z 
2022-11-23T01:46:52.4252470Z Installed:
2022-11-23T01:46:52.4252869Z   kernel-devel.x86_64 0:4.14.252-195.483.amzn2                                  
2022-11-23T01:46:52.4253087Z 
2022-11-23T01:46:52.4253204Z Complete!
2022-11-23T01:46:52.4598437Z + sudo modprobe backlight
2022-11-23T01:46:52.4885415Z + sudo curl -fsL -o /tmp/nvidia_driver https://s3.amazonaws.com/ossci-linux/nvidia_driver/NVIDIA-Linux-x86_64-515.76.run
2022-11-23T01:46:56.1714879Z + set +e
2022-11-23T01:46:56.1715448Z + sudo /bin/bash /tmp/nvidia_driver -s --no-drm
2022-11-23T01:46:57.5259044Z Verifying archive integrity... OK
2022-11-23T01:47:24.0082718Z Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 515.76................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
2022-11-23T01:47:24.1654483Z 
2022-11-23T01:47:24.1657850Z WARNING: The nvidia-drm module will not be installed. As a result, DRM-KMS will not function with this installation of the NVIDIA driver.
2022-11-23T01:47:24.1658600Z 
2022-11-23T01:47:35.6709319Z 
2022-11-23T01:47:35.6710643Z WARNING: nvidia-installer was forced to guess the X library path '/usr/lib64' and X module path '/usr/lib64/xorg/modules'; these paths were not queryable from the system.  If X fails to find the NVIDIA X driver module, please install the `pkg-config` utility and the X.Org SDK/development package for your distribution and reinstall the driver.
2022-11-23T01:47:35.6711645Z 
2022-11-23T01:47:43.0103765Z + NVIDIA_INSTALLATION_STATUS=0
2022-11-23T01:47:43.0104124Z + RESET_GPU=0
2022-11-23T01:47:43.0105884Z + '[' 0 -ne 0 ']'
2022-11-23T01:47:43.0106347Z ++ command -v nvidia-smi
2022-11-23T01:47:43.0107952Z + '[' -x /usr/bin/nvidia-smi ']'
2022-11-23T01:47:43.0112908Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0
2022-11-23T01:47:52.2749865Z + INSTALLED_DRIVER_VERSION=515.76
2022-11-23T01:47:52.2750308Z + NVIDIA_SMI_STATUS=0
2022-11-23T01:47:52.2750743Z + '[' 0 -ne 0 ']'
2022-11-23T01:47:52.2751058Z + '[' 0 -eq 1 ']'
2022-11-23T01:47:52.2752954Z + sudo rm -fv /tmp/nvidia_driver
2022-11-23T01:47:52.3457460Z removed ‘/tmp/nvidia_driver’
2022-11-23T01:47:52.3479554Z + set -e
2022-11-23T01:47:52.3480148Z + sudo modprobe nvidia
2022-11-23T01:47:52.3618870Z + echo 'After installing NVIDIA driver'
2022-11-23T01:47:52.3619181Z + lspci
2022-11-23T01:47:52.3619448Z After installing NVIDIA driver
2022-11-23T01:47:52.3883695Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
2022-11-23T01:47:52.3884229Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2022-11-23T01:47:52.3884645Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
2022-11-23T01:47:52.3885042Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01)
2022-11-23T01:47:52.3885612Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446
2022-11-23T01:47:52.3888566Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2022-11-23T01:47:52.3889303Z 00:1b.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2022-11-23T01:47:52.3889767Z 00:1c.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2022-11-23T01:47:52.3890201Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2022-11-23T01:47:52.3890605Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2022-11-23T01:47:52.3891037Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01)
2022-11-23T01:47:52.3891364Z + lsmod
2022-11-23T01:47:52.3910206Z Module                  Size  Used by
2022-11-23T01:47:52.3910490Z nvidia              40808448  0
2022-11-23T01:47:52.3910768Z drm                   425984  1 nvidia
2022-11-23T01:47:52.3911279Z i2c_core               77824  2 nvidia,drm
2022-11-23T01:47:52.3911551Z backlight              16384  0
2022-11-23T01:47:52.3911819Z xt_conntrack           16384  1
2022-11-23T01:47:52.3912091Z ipt_MASQUERADE         16384  1
2022-11-23T01:47:52.3912383Z nf_nat_masquerade_ipv4    16384  1 ipt_MASQUERADE
2022-11-23T01:47:52.3912710Z nf_conntrack_netlink    49152  0
2022-11-23T01:47:52.3913013Z nfnetlink              16384  2 nf_conntrack_netlink
2022-11-23T01:47:52.3913307Z xfrm_user              45056  1
2022-11-23T01:47:52.3913578Z xfrm_algo              16384  1 xfrm_user
2022-11-23T01:47:52.3913853Z xt_addrtype            16384  2
2022-11-23T01:47:52.3914118Z iptable_filter         16384  1
2022-11-23T01:47:52.3914360Z iptable_nat            16384  1
2022-11-23T01:47:52.3914622Z nf_conntrack_ipv4      16384  3
2022-11-23T01:47:52.3914974Z nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
2022-11-23T01:47:52.3915270Z nf_nat_ipv4            16384  1 iptable_nat
2022-11-23T01:47:52.3915593Z nf_nat                 36864  2 nf_nat_masquerade_ipv4,nf_nat_ipv4
2022-11-23T01:47:52.3916073Z nf_conntrack          155648  7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink
2022-11-23T01:47:52.3916474Z br_netfilter           24576  0
2022-11-23T01:47:52.3916741Z bridge                172032  1 br_netfilter
2022-11-23T01:47:52.3917030Z stp                    16384  1 bridge
2022-11-23T01:47:52.3917301Z llc                    16384  2 bridge,stp
2022-11-23T01:47:52.3917545Z overlay                86016  0
2022-11-23T01:47:52.3917793Z sunrpc                393216  1
2022-11-23T01:47:52.3918057Z dm_mirror              28672  0
2022-11-23T01:47:52.3922124Z dm_region_hash         20480  1 dm_mirror
2022-11-23T01:47:52.3922451Z dm_log                 20480  2 dm_region_hash,dm_mirror
2022-11-23T01:47:52.3922773Z dm_mod                143360  2 dm_log,dm_mirror
2022-11-23T01:47:52.3923064Z dax                    69632  1 dm_mod
2022-11-23T01:47:52.3923313Z sb_edac                24576  0
2022-11-23T01:47:52.3923571Z crc32_pclmul           16384  0
2022-11-23T01:47:52.3923845Z ghash_clmulni_intel    16384  0
2022-11-23T01:47:52.3924098Z pcbc                   16384  0
2022-11-23T01:47:52.3924365Z aesni_intel           188416  0
2022-11-23T01:47:52.3924626Z ata_piix               36864  0
2022-11-23T01:47:52.3924887Z aes_x86_64             20480  1 aesni_intel
2022-11-23T01:47:52.3925160Z crypto_simd            16384  1 aesni_intel
2022-11-23T01:47:52.3925453Z glue_helper            16384  1 aesni_intel
2022-11-23T01:47:52.3925741Z pcc_cpufreq            16384  0
2022-11-23T01:47:52.3925992Z libata                266240  1 ata_piix
2022-11-23T01:47:52.3926329Z cryptd                 28672  3 crypto_simd,ghash_clmulni_intel,aesni_intel
2022-11-23T01:47:52.3926653Z mousedev               24576  0
2022-11-23T01:47:52.3926889Z evdev                  20480  3
2022-11-23T01:47:52.3927144Z scsi_mod              245760  1 libata
2022-11-23T01:47:52.3927397Z psmouse                32768  0
2022-11-23T01:47:52.3927638Z button                 16384  0
2022-11-23T01:47:52.3927883Z ena                   114688  0
2022-11-23T01:47:52.3928136Z xen_blkfront           49152  2
2022-11-23T01:47:52.3928389Z crc32c_intel           24576  0
2022-11-23T01:47:52.3928834Z autofs4                49152  2
2022-11-23T01:47:52.3929126Z + modinfo nvidia
2022-11-23T01:47:52.3929620Z filename:       /lib/modules/4.14.252-195.483.amzn2.x86_64/kernel/drivers/video/nvidia.ko
2022-11-23T01:47:52.3930034Z firmware:       nvidia/515.76/gsp.bin
2022-11-23T01:47:52.3930378Z alias:          char-major-195-*
2022-11-23T01:47:52.3930645Z version:        515.76
2022-11-23T01:47:52.3930895Z supported:      external
2022-11-23T01:47:52.3931147Z license:        NVIDIA
2022-11-23T01:47:52.3931407Z srcversion:     51FD9DD90150B35351AFFBB
2022-11-23T01:47:52.3931705Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2022-11-23T01:47:52.3932018Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2022-11-23T01:47:52.3932420Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2022-11-23T01:47:52.3932741Z depends:        i2c-core,drm
2022-11-23T01:47:52.3933010Z retpoline:      Y
2022-11-23T01:47:52.3933255Z name:           nvidia
2022-11-23T01:47:52.3933643Z vermagic:       4.14.252-195.483.amzn2.x86_64 SMP mod_unload modversions 
2022-11-23T01:47:52.3934045Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2022-11-23T01:47:52.3934475Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2022-11-23T01:47:52.3934843Z parm:           NVreg_ResmanDebugLevel:int
2022-11-23T01:47:52.3935122Z parm:           NVreg_RmLogonRC:int
2022-11-23T01:47:52.3935430Z parm:           NVreg_ModifyDeviceFiles:int
2022-11-23T01:47:52.3935742Z parm:           NVreg_DeviceFileUID:int
2022-11-23T01:47:52.3936023Z parm:           NVreg_DeviceFileGID:int
2022-11-23T01:47:52.3936323Z parm:           NVreg_DeviceFileMode:int
2022-11-23T01:47:52.3936681Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2022-11-23T01:47:52.3937394Z parm:           NVreg_UsePageAttributeTable:int
2022-11-23T01:47:52.3937722Z parm:           NVreg_EnablePCIeGen3:int
2022-11-23T01:47:52.3938016Z parm:           NVreg_EnableMSI:int
2022-11-23T01:47:52.3938308Z parm:           NVreg_TCEBypassMode:int
2022-11-23T01:47:52.3938609Z parm:           NVreg_EnableStreamMemOPs:int
2022-11-23T01:47:52.3938972Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2022-11-23T01:47:52.3939362Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2022-11-23T01:47:52.3939717Z parm:           NVreg_EnableS0ixPowerManagement:int
2022-11-23T01:47:52.3940119Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2022-11-23T01:47:52.3940627Z parm:           NVreg_DynamicPowerManagement:int
2022-11-23T01:47:52.3941017Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2022-11-23T01:47:52.3941410Z parm:           NVreg_EnableGpuFirmware:int
2022-11-23T01:47:52.3941739Z parm:           NVreg_EnableGpuFirmwareLogs:int
2022-11-23T01:47:52.3942088Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2022-11-23T01:47:52.3942454Z parm:           NVreg_EnableUserNUMAManagement:int
2022-11-23T01:47:52.3942781Z parm:           NVreg_MemoryPoolSize:int
2022-11-23T01:47:52.3943097Z parm:           NVreg_KMallocHeapMaxSize:int
2022-11-23T01:47:52.3943403Z parm:           NVreg_VMallocHeapMaxSize:int
2022-11-23T01:47:52.3943716Z parm:           NVreg_IgnoreMMIOCheck:int
2022-11-23T01:47:52.3944015Z parm:           NVreg_NvLinkDisable:int
2022-11-23T01:47:52.3944444Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2022-11-23T01:47:52.3944797Z parm:           NVreg_RegisterPCIDriver:int
2022-11-23T01:47:52.3945120Z parm:           NVreg_EnableDbgBreakpoint:int
2022-11-23T01:47:52.3945421Z parm:           NVreg_RegistryDwords:charp
2022-11-23T01:47:52.3945758Z parm:           NVreg_RegistryDwordsPerDevice:charp
2022-11-23T01:47:52.3946073Z parm:           NVreg_RmMsg:charp
2022-11-23T01:47:52.3946345Z parm:           NVreg_GpuBlacklist:charp
2022-11-23T01:47:52.3946669Z parm:           NVreg_TemporaryFilePath:charp
2022-11-23T01:47:52.3946983Z parm:           NVreg_ExcludedGpus:charp
2022-11-23T01:47:52.3947275Z parm:           NVreg_DmaRemapPeerMmio:int
2022-11-23T01:47:52.3947630Z parm:           rm_firmware_active:charp
2022-11-23T01:47:52.3947972Z + set +e
2022-11-23T01:47:52.3948261Z + nvidia-smi
2022-11-23T01:47:59.4049902Z Wed Nov 23 01:47:59 2022       
2022-11-23T01:47:59.4050507Z +-----------------------------------------------------------------------------+
2022-11-23T01:47:59.4051072Z | NVIDIA-SMI 515.76       Driver Version: 515.76       CUDA Version: 11.7     |
2022-11-23T01:47:59.4051571Z |-------------------------------+----------------------+----------------------+
2022-11-23T01:47:59.4052067Z | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
2022-11-23T01:47:59.4052577Z | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
2022-11-23T01:47:59.4052944Z |                               |                      |               MIG M. |
2022-11-23T01:47:59.4053607Z |===============================+======================+======================|
2022-11-23T01:47:59.4877138Z |   0  Tesla M60           Off  | 00000000:00:1B.0 Off |          10516327738 |
2022-11-23T01:47:59.4877970Z | N/A   24C    P0    38W / 150W |      0MiB /  7680MiB |      0%      Default |
2022-11-23T01:47:59.4878708Z |                               |                      |                  N/A |
2022-11-23T01:47:59.4879740Z +-------------------------------+----------------------+----------------------+
2022-11-23T01:47:59.4947944Z |   1  Tesla M60           Off  | 00000000:00:1C.0 Off |                    0 |
2022-11-23T01:47:59.4948351Z | N/A   33C    P0    38W / 150W |      0MiB /  7680MiB |      0%      Default |
2022-11-23T01:47:59.4948707Z |                               |                      |                  N/A |
2022-11-23T01:47:59.4949185Z +-------------------------------+----------------------+----------------------+
2022-11-23T01:47:59.5048221Z |   2  Tesla M60           Off  | 00000000:00:1D.0 Off |                   68 |
2022-11-23T01:47:59.5048617Z | N/A   24C    P0    39W / 150W |      0MiB /  7680MiB |      0%      Default |
2022-11-23T01:47:59.5048988Z |                               |                      |                  N/A |
2022-11-23T01:47:59.5049468Z +-------------------------------+----------------------+----------------------+
2022-11-23T01:47:59.5098787Z |   3  Tesla M60           Off  | 00000000:00:1E.0 Off |                    0 |
2022-11-23T01:47:59.5099187Z | N/A   33C    P0    41W / 150W |      0MiB /  7680MiB |     55%      Default |
2022-11-23T01:47:59.5099530Z |                               |                      |                  N/A |
2022-11-23T01:47:59.5100009Z +-------------------------------+----------------------+----------------------+
2022-11-23T01:47:59.5100389Z                                                                                
2022-11-23T01:47:59.5100833Z +-----------------------------------------------------------------------------+
2022-11-23T01:47:59.5101222Z | Processes:                                                                  |
2022-11-23T01:47:59.5101595Z |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
2022-11-23T01:47:59.5101943Z |        ID   ID                                                   Usage      |
2022-11-23T01:47:59.5102254Z |=============================================================================|
2022-11-23T01:47:59.5113467Z |  No running processes found                                                 |
2022-11-23T01:47:59.5114049Z +-----------------------------------------------------------------------------+
2022-11-23T01:48:00.6098211Z + NVIDIA_SMI_STATUS=0
2022-11-23T01:48:00.6098757Z + '[' 0 -eq 0 ']'
2022-11-23T01:48:00.6099155Z + echo 'INFO: Ignoring allowed status 0'
2022-11-23T01:48:00.6099349Z + set -e
2022-11-23T01:48:00.6099682Z INFO: Ignoring allowed status 0
2022-11-23T01:48:00.6103677Z == Installing nvidia container toolkit for amzn2 ==
2022-11-23T01:48:00.6107828Z + sudo yum install -y yum-utils
2022-11-23T01:48:01.1780408Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-11-23T01:48:01.4560356Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version
2022-11-23T01:48:01.4560797Z Nothing to do
2022-11-23T01:48:01.4777979Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2022-11-23T01:48:02.0322330Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-11-23T01:48:02.0627688Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2022-11-23T01:48:02.0628512Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo
2022-11-23T01:48:02.0629058Z repo saved to /etc/yum.repos.d/nvidia-docker.repo
2022-11-23T01:48:02.0794591Z + sudo yum install -y nvidia-docker2
2022-11-23T01:48:02.6294317Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-11-23T01:48:02.6710709Z Retrieving key from https://nvidia.github.io/libnvidia-container/gpgkey
2022-11-23T01:48:02.6813273Z Importing GPG key 0xF796ECB0:
2022-11-23T01:48:02.6814159Z  Userid     : "NVIDIA CORPORATION (Open Source Projects) <cudatools@nvidia.com>"
2022-11-23T01:48:02.6814693Z  Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0
2022-11-23T01:48:02.6815119Z  From       : https://nvidia.github.io/libnvidia-container/gpgkey
2022-11-23T01:48:03.0962891Z Retrieving key from https://nvidia.github.io/nvidia-container-runtime/gpgkey
2022-11-23T01:48:03.1140441Z Importing GPG key 0xF796ECB0:
2022-11-23T01:48:03.1140884Z  Userid     : "NVIDIA CORPORATION (Open Source Projects) <cudatools@nvidia.com>"
2022-11-23T01:48:03.1141302Z  Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0
2022-11-23T01:48:03.1141726Z  From       : https://nvidia.github.io/nvidia-container-runtime/gpgkey
2022-11-23T01:48:03.3495209Z Retrieving key from https://nvidia.github.io/nvidia-docker/gpgkey
2022-11-23T01:48:03.3600268Z Importing GPG key 0xF796ECB0:
2022-11-23T01:48:03.3600892Z  Userid     : "NVIDIA CORPORATION (Open Source Projects) <cudatools@nvidia.com>"
2022-11-23T01:48:03.3601336Z  Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0
2022-11-23T01:48:03.3601871Z  From       : https://nvidia.github.io/nvidia-docker/gpgkey
2022-11-23T01:48:05.1616639Z Resolving Dependencies
2022-11-23T01:48:05.1625136Z --> Running transaction check
2022-11-23T01:48:05.1625702Z ---> Package nvidia-docker2.noarch 0:2.11.0-1 will be installed
2022-11-23T01:48:05.1652678Z --> Processing Dependency: nvidia-container-toolkit >= 1.10.0-1 for package: nvidia-docker2-2.11.0-1.noarch
2022-11-23T01:48:05.2037509Z --> Running transaction check
2022-11-23T01:48:05.2038123Z ---> Package nvidia-container-toolkit.x86_64 0:1.11.0-1 will be installed
2022-11-23T01:48:05.2199995Z --> Processing Dependency: nvidia-container-toolkit-base = 1.11.0-1 for package: nvidia-container-toolkit-1.11.0-1.x86_64
2022-11-23T01:48:05.2210900Z --> Processing Dependency: libnvidia-container-tools < 2.0.0 for package: nvidia-container-toolkit-1.11.0-1.x86_64
2022-11-23T01:48:05.2343369Z --> Processing Dependency: libnvidia-container-tools >= 1.11.0-1 for package: nvidia-container-toolkit-1.11.0-1.x86_64
2022-11-23T01:48:05.2343963Z --> Running transaction check
2022-11-23T01:48:05.2344316Z ---> Package libnvidia-container-tools.x86_64 0:1.11.0-1 will be installed
2022-11-23T01:48:05.2355243Z --> Processing Dependency: libnvidia-container1(x86-64) >= 1.11.0-1 for package: libnvidia-container-tools-1.11.0-1.x86_64
2022-11-23T01:48:05.2383768Z --> Processing Dependency: libnvidia-container.so.1(NVC_1.0)(64bit) for package: libnvidia-container-tools-1.11.0-1.x86_64
2022-11-23T01:48:05.2384557Z --> Processing Dependency: libnvidia-container.so.1()(64bit) for package: libnvidia-container-tools-1.11.0-1.x86_64
2022-11-23T01:48:05.2385204Z ---> Package nvidia-container-toolkit-base.x86_64 0:1.11.0-1 will be installed
2022-11-23T01:48:05.2385847Z --> Running transaction check
2022-11-23T01:48:05.2386557Z ---> Package libnvidia-container1.x86_64 0:1.11.0-1 will be installed
2022-11-23T01:48:05.5392402Z --> Finished Dependency Resolution
2022-11-23T01:48:05.6170465Z 
2022-11-23T01:48:05.6170903Z Dependencies Resolved
2022-11-23T01:48:05.6187151Z 
2022-11-23T01:48:05.6187738Z ================================================================================
2022-11-23T01:48:05.6188678Z  Package                        Arch    Version      Repository            Size
2022-11-23T01:48:05.6189358Z ================================================================================
2022-11-23T01:48:05.6189631Z Installing:
2022-11-23T01:48:05.6190094Z  nvidia-docker2                 noarch  2.11.0-1     libnvidia-container  8.7 k
2022-11-23T01:48:05.6190424Z Installing for dependencies:
2022-11-23T01:48:05.6190809Z  libnvidia-container-tools      x86_64  1.11.0-1     libnvidia-container   49 k
2022-11-23T01:48:05.6191550Z  libnvidia-container1           x86_64  1.11.0-1     libnvidia-container  1.0 M
2022-11-23T01:48:05.6192058Z  nvidia-container-toolkit       x86_64  1.11.0-1     libnvidia-container  780 k
2022-11-23T01:48:05.6192602Z  nvidia-container-toolkit-base  x86_64  1.11.0-1     libnvidia-container  2.5 M
2022-11-23T01:48:05.6192850Z 
2022-11-23T01:48:05.6192966Z Transaction Summary
2022-11-23T01:48:05.6193263Z ================================================================================
2022-11-23T01:48:05.6193585Z Install  1 Package (+4 Dependent packages)
2022-11-23T01:48:05.6193841Z 
2022-11-23T01:48:05.6193957Z Total download size: 4.3 M
2022-11-23T01:48:05.6194241Z Installed size: 12 M
2022-11-23T01:48:05.6194508Z Downloading packages:
2022-11-23T01:48:05.7594562Z --------------------------------------------------------------------------------
2022-11-23T01:48:05.7595025Z Total                                               31 MB/s | 4.3 MB  00:00     
2022-11-23T01:48:05.7648368Z Running transaction check
2022-11-23T01:48:05.7823793Z Running transaction test
2022-11-23T01:48:05.7998126Z Transaction test succeeded
2022-11-23T01:48:05.7998815Z Running transaction
2022-11-23T01:48:06.3187920Z   Installing : nvidia-container-toolkit-base-1.11.0-1.x86_64                1/5 
2022-11-23T01:48:06.3602502Z   Installing : libnvidia-container1-1.11.0-1.x86_64                         2/5 
2022-11-23T01:48:06.4782748Z   Installing : libnvidia-container-tools-1.11.0-1.x86_64                    3/5 
2022-11-23T01:48:06.5075381Z   Installing : nvidia-container-toolkit-1.11.0-1.x86_64                     4/5 
2022-11-23T01:48:06.5534499Z   Installing : nvidia-docker2-2.11.0-1.noarch                               5/5 
2022-11-23T01:48:06.5669497Z   Verifying  : libnvidia-container1-1.11.0-1.x86_64                         1/5 
2022-11-23T01:48:06.5789143Z   Verifying  : nvidia-container-toolkit-base-1.11.0-1.x86_64                2/5 
2022-11-23T01:48:06.5912065Z   Verifying  : nvidia-container-toolkit-1.11.0-1.x86_64                     3/5 
2022-11-23T01:48:06.6023052Z   Verifying  : libnvidia-container-tools-1.11.0-1.x86_64                    4/5 
2022-11-23T01:48:06.6864417Z   Verifying  : nvidia-docker2-2.11.0-1.noarch                               5/5 
2022-11-23T01:48:06.6864644Z 
2022-11-23T01:48:06.6864737Z Installed:
2022-11-23T01:48:06.6865203Z   nvidia-docker2.noarch 0:2.11.0-1                                              
2022-11-23T01:48:06.6865484Z 
2022-11-23T01:48:06.6865613Z Dependency Installed:
2022-11-23T01:48:06.6867812Z   libnvidia-container-tools.x86_64 0:1.11.0-1                                   
2022-11-23T01:48:06.6868379Z   libnvidia-container1.x86_64 0:1.11.0-1                                        
2022-11-23T01:48:06.6868859Z   nvidia-container-toolkit.x86_64 0:1.11.0-1                                    
2022-11-23T01:48:06.6869356Z   nvidia-container-toolkit-base.x86_64 0:1.11.0-1                               
2022-11-23T01:48:06.6869610Z 
2022-11-23T01:48:06.6869733Z Complete!
2022-11-23T01:48:06.7901687Z + sudo systemctl restart docker
2022-11-23T01:48:14.6057883Z Command completed after 1 attempt(s).
2022-11-23T01:48:14.6058123Z 
2022-11-23T01:48:14.6060776Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-11-23T01:48:14.6116108Z ##[group]Run python3 -m pip install psutil==5.9.1
2022-11-23T01:48:14.6116665Z python3 -m pip install psutil==5.9.1
2022-11-23T01:48:14.6116963Z python3 -m pip install pynvml==11.4.1
2022-11-23T01:48:14.6117310Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 &
2022-11-23T01:48:14.6117686Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}"
2022-11-23T01:48:14.6131880Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:48:14.6132168Z env:
2022-11-23T01:48:14.6132540Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:48:14.6132790Z   GPU_FLAG: --gpus all
2022-11-23T01:48:14.6133044Z ##[endgroup]
2022-11-23T01:48:15.3657617Z Defaulting to user installation because normal site-packages is not writeable
2022-11-23T01:48:15.7644002Z Collecting psutil==5.9.1
2022-11-23T01:48:15.7941506Z   Downloading psutil-5.9.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB)
2022-11-23T01:48:15.8691633Z Installing collected packages: psutil
2022-11-23T01:48:16.0324204Z Successfully installed psutil-5.9.1
2022-11-23T01:48:16.5368753Z Defaulting to user installation because normal site-packages is not writeable
2022-11-23T01:48:16.6238938Z Collecting pynvml==11.4.1
2022-11-23T01:48:16.6394105Z   Downloading pynvml-11.4.1-py3-none-any.whl (46 kB)
2022-11-23T01:48:16.6894849Z Installing collected packages: pynvml
2022-11-23T01:48:16.7464003Z Successfully installed pynvml-11.4.1
2022-11-23T01:48:16.8113439Z Prepare all required actions
2022-11-23T01:48:16.8114015Z Getting action download info
2022-11-23T01:48:17.3705647Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:4a8bfae15cc25cc0785c1603ee87a9da8fd442ea)
2022-11-23T01:48:17.5534751Z Download action repository 'actions/download-artifact@v3' (SHA:9782bd6a9848b53b110e712e20e42d89988822b7)
2022-11-23T01:48:17.6764033Z ##[group]Run ./.github/actions/download-build-artifacts
2022-11-23T01:48:17.6764359Z with:
2022-11-23T01:48:17.6764655Z   name: linux-bionic-cuda11.6-py3.9-gcc7
2022-11-23T01:48:17.6764927Z env:
2022-11-23T01:48:17.6765180Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:48:17.6765459Z   GPU_FLAG: --gpus all
2022-11-23T01:48:17.6765695Z ##[endgroup]
2022-11-23T01:48:17.6796402Z ##[group]Run seemethere/download-artifact-s3@v4
2022-11-23T01:48:17.6796705Z with:
2022-11-23T01:48:17.6796972Z   name: linux-bionic-cuda11.6-py3.9-gcc7
2022-11-23T01:48:17.6797290Z   s3-bucket: gha-artifacts
2022-11-23T01:48:17.6797586Z   region: us-east-1
2022-11-23T01:48:17.6797823Z env:
2022-11-23T01:48:17.6798084Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:48:17.6798337Z   GPU_FLAG: --gpus all
2022-11-23T01:48:17.6798594Z ##[endgroup]
2022-11-23T01:48:18.2244020Z Found 1 objects with prefix pytorch/pytorch/3528394938/linux-bionic-cuda11.6-py3.9-gcc7/
2022-11-23T01:48:18.2244640Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-11-23T01:48:33.5675792Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-11-23T01:48:33.5676323Z 
2022-11-23T01:48:33.5684470Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-11-23T01:48:33.5686900Z Artifact download has finished successfully
2022-11-23T01:48:33.6021117Z ##[group]Run unzip -o artifacts.zip
2022-11-23T01:48:33.6021448Z unzip -o artifacts.zip
2022-11-23T01:48:33.6037523Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:48:33.6037994Z env:
2022-11-23T01:48:33.6038242Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:48:33.6038499Z   GPU_FLAG: --gpus all
2022-11-23T01:48:33.6038749Z ##[endgroup]
2022-11-23T01:48:33.6114057Z Archive:  artifacts.zip
2022-11-23T01:48:33.6115054Z    creating: dist/
2022-11-23T01:48:35.7578037Z   inflating: dist/torch-1.14.0a0+git1cfd385-cp310-cp310-linux_x86_64.whl  
2022-11-23T01:48:35.7578558Z    creating: build/custom_test_artifacts/
2022-11-23T01:48:35.7579004Z    creating: build/custom_test_artifacts/custom-op-build/
2022-11-23T01:48:35.7579485Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2022-11-23T01:48:35.7586300Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeOutput.log  
2022-11-23T01:48:35.7586885Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/
2022-11-23T01:48:35.7587478Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-11-23T01:48:35.7588320Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/
2022-11-23T01:48:35.7588883Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-11-23T01:48:35.7589608Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-11-23T01:48:35.7591015Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-11-23T01:48:35.7591592Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-11-23T01:48:35.7592149Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-11-23T01:48:35.7594647Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-11-23T01:48:35.7596052Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-11-23T01:48:35.7597417Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-11-23T01:48:35.7598107Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-11-23T01:48:35.7599663Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-11-23T01:48:35.7600609Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-11-23T01:48:35.7601205Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-11-23T01:48:35.7601766Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-11-23T01:48:35.7656789Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-11-23T01:48:35.7657985Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-11-23T01:48:35.7658729Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-11-23T01:48:35.7659461Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-11-23T01:48:35.7660194Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-11-23T01:48:35.7660898Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-11-23T01:48:35.7661608Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2022-11-23T01:48:35.7662315Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-11-23T01:48:35.7663016Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-11-23T01:48:35.7704818Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-11-23T01:48:35.7746499Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-11-23T01:48:35.7747244Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-11-23T01:48:35.7747914Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2022-11-23T01:48:35.7748575Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-11-23T01:48:35.7749235Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-11-23T01:48:35.7750079Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-11-23T01:48:35.7750726Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-11-23T01:48:35.7752234Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-11-23T01:48:35.7828048Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-11-23T01:48:35.7903541Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-11-23T01:48:35.7904224Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-11-23T01:48:35.7904782Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2022-11-23T01:48:35.7905496Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeError.log  
2022-11-23T01:48:35.7906086Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2022-11-23T01:48:35.7906615Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2022-11-23T01:48:35.7907209Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2022-11-23T01:48:35.7907823Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2022-11-23T01:48:35.7908427Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2022-11-23T01:48:35.7908989Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2022-11-23T01:48:35.7909581Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2022-11-23T01:48:35.7910174Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2022-11-23T01:48:35.7910775Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2022-11-23T01:48:35.7911348Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2022-11-23T01:48:35.7911941Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2022-11-23T01:48:35.7932137Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2022-11-23T01:48:35.8047304Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2022-11-23T01:48:35.8047899Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2022-11-23T01:48:35.8048901Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2022-11-23T01:48:35.8049590Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2022-11-23T01:48:35.8050220Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2022-11-23T01:48:35.8050900Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2022-11-23T01:48:35.8051522Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2022-11-23T01:48:35.8052129Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2022-11-23T01:48:35.8052746Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2022-11-23T01:48:35.8053336Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2022-11-23T01:48:35.8053940Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2022-11-23T01:48:35.8074216Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2022-11-23T01:48:35.8160274Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2022-11-23T01:48:35.8160938Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-11-23T01:48:35.8161563Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2022-11-23T01:48:35.8162141Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2022-11-23T01:48:35.8162698Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2022-11-23T01:48:35.8163220Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2022-11-23T01:48:35.8163770Z   inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc  
2022-11-23T01:48:35.8166091Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2022-11-23T01:48:35.8166784Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2022-11-23T01:48:35.8167693Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2022-11-23T01:48:35.8263780Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2022-11-23T01:48:35.8329550Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2022-11-23T01:48:35.8330051Z    creating: build/custom_test_artifacts/jit-hook-build/
2022-11-23T01:48:35.8330532Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2022-11-23T01:48:35.8337317Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeOutput.log  
2022-11-23T01:48:35.8338290Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/
2022-11-23T01:48:35.8338845Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-11-23T01:48:35.8339421Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/
2022-11-23T01:48:35.8339988Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-11-23T01:48:35.8340747Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-11-23T01:48:35.8342118Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-11-23T01:48:35.8342692Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-11-23T01:48:35.8343262Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-11-23T01:48:35.8345516Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-11-23T01:48:35.8346690Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-11-23T01:48:35.8348149Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-11-23T01:48:35.8349086Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-11-23T01:48:35.8350393Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-11-23T01:48:35.8351290Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-11-23T01:48:35.8351885Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-11-23T01:48:35.8352464Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-11-23T01:48:35.8408668Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-11-23T01:48:35.8409428Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-11-23T01:48:35.8410173Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-11-23T01:48:35.8411124Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-11-23T01:48:35.8411836Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-11-23T01:48:35.8412540Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-11-23T01:48:35.8413245Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2022-11-23T01:48:35.8413938Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-11-23T01:48:35.8414706Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-11-23T01:48:35.8455728Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-11-23T01:48:35.8497363Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-11-23T01:48:35.8499170Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-11-23T01:48:35.8499889Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2022-11-23T01:48:35.8500503Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-11-23T01:48:35.8501163Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-11-23T01:48:35.8501796Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-11-23T01:48:35.8502433Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-11-23T01:48:35.8503441Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-11-23T01:48:35.8577332Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-11-23T01:48:35.8650706Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-11-23T01:48:35.8651397Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-11-23T01:48:35.8651933Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2022-11-23T01:48:35.8652487Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeError.log  
2022-11-23T01:48:35.8653052Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2022-11-23T01:48:35.8653611Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2022-11-23T01:48:35.8654191Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2022-11-23T01:48:35.8654832Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2022-11-23T01:48:35.8655445Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2022-11-23T01:48:35.8656035Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2022-11-23T01:48:35.8656610Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2022-11-23T01:48:35.8658100Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2022-11-23T01:48:35.8658998Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2022-11-23T01:48:35.8659783Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2022-11-23T01:48:35.8660362Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2022-11-23T01:48:35.8680834Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2022-11-23T01:48:35.8747424Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2022-11-23T01:48:35.8748118Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-11-23T01:48:35.8748723Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2022-11-23T01:48:35.8749288Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2022-11-23T01:48:35.8749996Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2022-11-23T01:48:35.8750569Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2022-11-23T01:48:35.8751092Z   inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc  
2022-11-23T01:48:35.8753019Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2022-11-23T01:48:35.8753701Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2022-11-23T01:48:35.8754625Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2022-11-23T01:48:35.8804492Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2022-11-23T01:48:35.8805023Z    creating: build/custom_test_artifacts/custom-backend-build/
2022-11-23T01:48:35.8805539Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2022-11-23T01:48:35.8812105Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeOutput.log  
2022-11-23T01:48:35.8812699Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/
2022-11-23T01:48:35.8813299Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-11-23T01:48:35.8813872Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/
2022-11-23T01:48:35.8814470Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-11-23T01:48:35.8815563Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-11-23T01:48:35.8817200Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-11-23T01:48:35.8817818Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-11-23T01:48:35.8818394Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-11-23T01:48:35.8820702Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-11-23T01:48:35.8821897Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-11-23T01:48:35.8823390Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-11-23T01:48:35.8824075Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-11-23T01:48:35.8825556Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-11-23T01:48:35.8826509Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-11-23T01:48:35.8827126Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-11-23T01:48:35.8827707Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-11-23T01:48:35.8882507Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-11-23T01:48:35.8883496Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-11-23T01:48:35.8884270Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-11-23T01:48:35.8885030Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-11-23T01:48:35.8885790Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-11-23T01:48:35.8886520Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-11-23T01:48:35.8887342Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2022-11-23T01:48:35.8888070Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-11-23T01:48:35.8888799Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-11-23T01:48:35.8930418Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-11-23T01:48:35.8971930Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-11-23T01:48:35.8972718Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-11-23T01:48:35.8973399Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2022-11-23T01:48:35.8974086Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-11-23T01:48:35.8974762Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-11-23T01:48:35.8975435Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-11-23T01:48:35.8976356Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-11-23T01:48:35.8978607Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-11-23T01:48:35.9052042Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-11-23T01:48:35.9126815Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-11-23T01:48:35.9127534Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-11-23T01:48:35.9128117Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2022-11-23T01:48:35.9128693Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeError.log  
2022-11-23T01:48:35.9129278Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2022-11-23T01:48:35.9129866Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2022-11-23T01:48:35.9130486Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2022-11-23T01:48:35.9131154Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2022-11-23T01:48:35.9131794Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2022-11-23T01:48:35.9132441Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2022-11-23T01:48:35.9133227Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2022-11-23T01:48:35.9133868Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2022-11-23T01:48:35.9134514Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2022-11-23T01:48:35.9135173Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2022-11-23T01:48:35.9135787Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2022-11-23T01:48:35.9139078Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2022-11-23T01:48:35.9288846Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2022-11-23T01:48:35.9289556Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2022-11-23T01:48:35.9290196Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2022-11-23T01:48:35.9290889Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2022-11-23T01:48:35.9291550Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2022-11-23T01:48:35.9292195Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2022-11-23T01:48:35.9292830Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2022-11-23T01:48:35.9293491Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2022-11-23T01:48:35.9294158Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2022-11-23T01:48:35.9294820Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2022-11-23T01:48:35.9295447Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2022-11-23T01:48:35.9315271Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2022-11-23T01:48:35.9374044Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2022-11-23T01:48:35.9374768Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-11-23T01:48:35.9375410Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2022-11-23T01:48:35.9376027Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2022-11-23T01:48:35.9376603Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2022-11-23T01:48:35.9377910Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2022-11-23T01:48:35.9378869Z   inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc  
2022-11-23T01:48:35.9380635Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2022-11-23T01:48:35.9381589Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2022-11-23T01:48:35.9382534Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2022-11-23T01:48:35.9503499Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2022-11-23T01:48:35.9551252Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2022-11-23T01:48:35.9551667Z    creating: build/lib/
2022-11-23T01:48:35.9552143Z   inflating: build/lib/libclog.a     
2022-11-23T01:48:35.9621120Z   inflating: build/lib/libgtest.a    
2022-11-23T01:48:35.9631374Z   inflating: build/lib/libpthreadpool.a  
2022-11-23T01:48:35.9641107Z   inflating: build/lib/libittnotify.a  
2022-11-23T01:48:35.9750787Z   inflating: build/lib/libprotobuf-lite.a  
2022-11-23T01:48:35.9845495Z   inflating: build/lib/libbenchmark.a  
2022-11-23T01:48:35.9878098Z   inflating: build/lib/libtensorpipe_uv.a  
2022-11-23T01:48:36.0014322Z   inflating: build/lib/libgloo.a     
2022-11-23T01:48:36.0569392Z   inflating: build/lib/libprotobuf.a  
2022-11-23T01:48:36.0649017Z   inflating: build/lib/libasmjit.a   
2022-11-23T01:48:36.0682010Z   inflating: build/lib/libfmt.a      
2022-11-23T01:48:36.0682354Z   inflating: build/lib/libfoxi_loader.a  
2022-11-23T01:48:36.0684081Z   inflating: build/lib/libcaffe2_nvrtc.so  
2022-11-23T01:48:36.0767738Z   inflating: build/lib/libc10.so     
2022-11-23T01:48:36.0768256Z   inflating: build/lib/libtorch_global_deps.so  
2022-11-23T01:48:36.0779046Z   inflating: build/lib/libcpuinfo.a  
2022-11-23T01:48:36.0788325Z   inflating: build/lib/libcpuinfo_internals.a  
2022-11-23T01:48:36.0789714Z   inflating: build/lib/libnnpack_reference_layers.a  
2022-11-23T01:48:36.1383421Z   inflating: build/lib/libprotoc.a   
2022-11-23T01:48:36.1403282Z   inflating: build/lib/libgmock.a    
2022-11-23T01:48:36.1403634Z   inflating: build/lib/libgtest_main.a  
2022-11-23T01:48:36.1403953Z   inflating: build/lib/libbenchmark_main.a  
2022-11-23T01:48:36.1553630Z   inflating: build/lib/libXNNPACK.a  
2022-11-23T01:48:37.1659550Z   inflating: build/lib/libdnnl.a     
2022-11-23T01:48:37.2331924Z   inflating: build/lib/libtensorpipe.a  
2022-11-23T01:48:37.2385742Z   inflating: build/lib/libc10_cuda.so  
2022-11-23T01:48:37.2402179Z   inflating: build/lib/libqnnpack.a  
2022-11-23T01:48:37.3986965Z   inflating: build/lib/libfbgemm.a   
2022-11-23T01:48:37.3987432Z   inflating: build/lib/libgmock_main.a  
2022-11-23T01:48:37.4011350Z   inflating: build/lib/libpytorch_qnnpack.a  
2022-11-23T01:48:37.5187087Z   inflating: build/lib/libdnnl_graph.a  
2022-11-23T01:48:37.5483947Z   inflating: build/lib/libtensorpipe_cuda.a  
2022-11-23T01:48:37.6012218Z   inflating: build/lib/libkineto.a   
2022-11-23T01:48:37.6058860Z   inflating: build/lib/libcaffe2_protos.a  
2022-11-23T01:48:37.6106554Z   inflating: build/lib/libonnx_proto.a  
2022-11-23T01:48:37.6129388Z   inflating: build/lib/libnnpack.a   
2022-11-23T01:48:37.6824129Z   inflating: build/lib/libonnx.a     
2022-11-23T01:48:37.7271484Z   inflating: build/lib/libgloo_cuda.a  
2022-11-23T01:48:40.1570320Z   inflating: build/lib/libtorch_cpu.so  
2022-11-23T01:48:42.3551026Z   inflating: build/lib/libtorch_cuda.so  
2022-11-23T01:48:42.3551413Z   inflating: build/lib/libtorch.so   
2022-11-23T01:48:42.3553730Z   inflating: build/lib/libc10d_cuda_test.so  
2022-11-23T01:48:43.3704112Z   inflating: build/lib/libtorch_cuda_linalg.so  
2022-11-23T01:48:43.3727483Z   inflating: build/lib/libjitbackend_test.so  
2022-11-23T01:48:43.3789724Z   inflating: build/lib/libtorchbind_test.so  
2022-11-23T01:48:43.3821219Z   inflating: build/lib/libbackend_with_compiler.so  
2022-11-23T01:48:43.3825644Z   inflating: build/lib/libshm.so     
2022-11-23T01:48:43.5680932Z   inflating: build/lib/libtorch_python.so  
2022-11-23T01:48:43.5722326Z   inflating: build/lib/libnnapi_backend.so  
2022-11-23T01:48:43.5722966Z    creating: build/bin/
2022-11-23T01:48:43.5776350Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2022-11-23T01:48:43.5832420Z   inflating: build/bin/c10_DeviceGuard_test  
2022-11-23T01:48:43.5887023Z   inflating: build/bin/c10_Device_test  
2022-11-23T01:48:43.5952446Z   inflating: build/bin/c10_DispatchKeySet_test  
2022-11-23T01:48:43.6005285Z   inflating: build/bin/c10_StreamGuard_test  
2022-11-23T01:48:43.6059086Z   inflating: build/bin/c10_SymInt_test  
2022-11-23T01:48:43.6119661Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2022-11-23T01:48:43.6180094Z   inflating: build/bin/c10_InlineStreamGuard_test  
2022-11-23T01:48:43.6242964Z   inflating: build/bin/c10_SizesAndStrides_test  
2022-11-23T01:48:43.6295929Z   inflating: build/bin/c10_Array_test  
2022-11-23T01:48:43.6354328Z   inflating: build/bin/c10_Bitset_test  
2022-11-23T01:48:43.6410875Z   inflating: build/bin/c10_C++17_test  
2022-11-23T01:48:43.6463208Z   inflating: build/bin/c10_ConstexprCrc_test  
2022-11-23T01:48:43.6517204Z   inflating: build/bin/c10_DeadlockDetection_test  
2022-11-23T01:48:43.6571213Z   inflating: build/bin/c10_Half_test  
2022-11-23T01:48:43.6634455Z   inflating: build/bin/c10_LeftRight_test  
2022-11-23T01:48:43.6702952Z   inflating: build/bin/c10_Metaprogramming_test  
2022-11-23T01:48:43.6862308Z   inflating: build/bin/c10_SmallVectorTest  
2022-11-23T01:48:43.6917904Z   inflating: build/bin/c10_Synchronized_test  
2022-11-23T01:48:43.6980927Z   inflating: build/bin/c10_ThreadLocal_test  
2022-11-23T01:48:43.7038363Z   inflating: build/bin/c10_TypeIndex_test  
2022-11-23T01:48:43.7093389Z   inflating: build/bin/c10_TypeList_test  
2022-11-23T01:48:43.7146218Z   inflating: build/bin/c10_TypeTraits_test  
2022-11-23T01:48:43.7203233Z   inflating: build/bin/c10_accumulate_test  
2022-11-23T01:48:43.7264863Z   inflating: build/bin/c10_bfloat16_test  
2022-11-23T01:48:43.7326022Z   inflating: build/bin/c10_complex_math_test  
2022-11-23T01:48:43.7386776Z   inflating: build/bin/c10_complex_test  
2022-11-23T01:48:43.7508832Z   inflating: build/bin/c10_either_test  
2022-11-23T01:48:43.7566771Z   inflating: build/bin/c10_exception_test  
2022-11-23T01:48:43.7622568Z   inflating: build/bin/c10_flags_test  
2022-11-23T01:48:43.7810284Z   inflating: build/bin/c10_intrusive_ptr_test  
2022-11-23T01:48:43.7866275Z   inflating: build/bin/c10_irange_test  
2022-11-23T01:48:43.7930748Z   inflating: build/bin/c10_logging_test  
2022-11-23T01:48:43.8012315Z   inflating: build/bin/c10_optional_test  
2022-11-23T01:48:43.8072358Z   inflating: build/bin/c10_registry_test  
2022-11-23T01:48:43.8141559Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2022-11-23T01:48:43.8198303Z   inflating: build/bin/c10_tempfile_test  
2022-11-23T01:48:43.8263000Z   inflating: build/bin/c10_string_view_test  
2022-11-23T01:48:43.8325277Z   inflating: build/bin/c10_typeid_test  
2022-11-23T01:48:43.8387368Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2022-11-23T01:48:43.8926201Z   inflating: build/bin/protoc-3.13.0.0  
2022-11-23T01:48:43.9462373Z   inflating: build/bin/protoc        
2022-11-23T01:48:43.9516596Z   inflating: build/bin/c10_cuda_CUDATest  
2022-11-23T01:48:43.9880133Z   inflating: build/bin/vec_test_all_types_AVX2  
2022-11-23T01:48:44.0206082Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2022-11-23T01:48:44.0265006Z   inflating: build/bin/FileStoreTest  
2022-11-23T01:48:44.0324483Z   inflating: build/bin/HashStoreTest  
2022-11-23T01:48:44.0390856Z   inflating: build/bin/TCPStoreTest  
2022-11-23T01:48:44.0406314Z   inflating: build/bin/ProcessGroupMPITest  
2022-11-23T01:48:44.0408632Z   inflating: build/bin/example_allreduce  
2022-11-23T01:48:44.0467717Z   inflating: build/bin/Dimname_test  
2022-11-23T01:48:44.0547383Z   inflating: build/bin/Dict_test     
2022-11-23T01:48:44.0617567Z   inflating: build/bin/MaybeOwned_test  
2022-11-23T01:48:44.0679605Z   inflating: build/bin/NamedTensor_test  
2022-11-23T01:48:44.0744820Z   inflating: build/bin/atest         
2022-11-23T01:48:44.0809584Z   inflating: build/bin/apply_utils_test  
2022-11-23T01:48:44.0877156Z   inflating: build/bin/basic         
2022-11-23T01:48:44.0936318Z   inflating: build/bin/broadcast_test  
2022-11-23T01:48:44.1001081Z   inflating: build/bin/cpu_generator_test  
2022-11-23T01:48:44.1057654Z   inflating: build/bin/cpu_profiling_allocator_test  
2022-11-23T01:48:44.1112505Z   inflating: build/bin/dispatch_key_set_test  
2022-11-23T01:48:44.1209202Z   inflating: build/bin/cpu_rng_test  
2022-11-23T01:48:44.1263657Z   inflating: build/bin/dlconvertor_test  
2022-11-23T01:48:44.1327551Z   inflating: build/bin/extension_backend_test  
2022-11-23T01:48:44.1388096Z   inflating: build/bin/half_test     
2022-11-23T01:48:44.1492538Z   inflating: build/bin/ivalue_test   
2022-11-23T01:48:44.1546987Z   inflating: build/bin/lazy_tensor_test  
2022-11-23T01:48:44.1606339Z   inflating: build/bin/memory_format_test  
2022-11-23T01:48:44.1664647Z   inflating: build/bin/math_kernel_test  
2022-11-23T01:48:44.1723908Z   inflating: build/bin/memory_overlapping_test  
2022-11-23T01:48:44.1778943Z   inflating: build/bin/operator_name_test  
2022-11-23T01:48:44.1836564Z   inflating: build/bin/mobile_memory_cleanup  
2022-11-23T01:48:44.1897750Z   inflating: build/bin/native_test   
2022-11-23T01:48:44.1952196Z   inflating: build/bin/operators_test  
2022-11-23T01:48:44.2010107Z   inflating: build/bin/packedtensoraccessor_test  
2022-11-23T01:48:44.2081564Z   inflating: build/bin/pow_test      
2022-11-23T01:48:44.2145129Z   inflating: build/bin/quantized_test  
2022-11-23T01:48:44.2200914Z   inflating: build/bin/reportMemoryUsage_test  
2022-11-23T01:48:44.2254310Z   inflating: build/bin/reduce_ops_test  
2022-11-23T01:48:44.2317021Z   inflating: build/bin/scalar_tensor_test  
2022-11-23T01:48:44.2379341Z   inflating: build/bin/scalar_test   
2022-11-23T01:48:44.2436084Z   inflating: build/bin/stride_properties_test  
2022-11-23T01:48:44.2522676Z   inflating: build/bin/tensor_iterator_test  
2022-11-23T01:48:44.2583168Z   inflating: build/bin/type_ptr_test  
2022-11-23T01:48:44.2585035Z   inflating: build/bin/thread_init_test  
2022-11-23T01:48:44.2647498Z   inflating: build/bin/test_parallel  
2022-11-23T01:48:44.2700957Z   inflating: build/bin/variant_test  
2022-11-23T01:48:44.2767228Z   inflating: build/bin/type_test     
2022-11-23T01:48:44.2824919Z   inflating: build/bin/undefined_tensor_test  
2022-11-23T01:48:44.2825430Z   inflating: build/bin/verify_api_visibility  
2022-11-23T01:48:44.2902062Z   inflating: build/bin/vmap_test     
2022-11-23T01:48:44.2957942Z   inflating: build/bin/weakref_test  
2022-11-23T01:48:44.3023414Z   inflating: build/bin/IListRef_test  
2022-11-23T01:48:44.3077139Z   inflating: build/bin/xla_tensor_test  
2022-11-23T01:48:44.3132960Z   inflating: build/bin/wrapdim_test  
2022-11-23T01:48:44.3253135Z   inflating: build/bin/List_test     
2022-11-23T01:48:44.3387747Z   inflating: build/bin/kernel_function_legacy_test  
2022-11-23T01:48:44.3494316Z   inflating: build/bin/kernel_function_test  
2022-11-23T01:48:44.3566102Z   inflating: build/bin/KernelFunction_test  
2022-11-23T01:48:44.3707120Z   inflating: build/bin/kernel_lambda_legacy_test  
2022-11-23T01:48:44.3822730Z   inflating: build/bin/kernel_lambda_test  
2022-11-23T01:48:44.3887518Z   inflating: build/bin/kernel_stackbased_test  
2022-11-23T01:48:44.3942970Z   inflating: build/bin/CppSignature_test  
2022-11-23T01:48:44.4048686Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2022-11-23T01:48:44.4100659Z   inflating: build/bin/op_allowlist_test  
2022-11-23T01:48:44.4422523Z   inflating: build/bin/op_registration_test  
2022-11-23T01:48:44.4480311Z   inflating: build/bin/inline_container_test  
2022-11-23T01:48:44.4543351Z   inflating: build/bin/backend_fallback_test  
2022-11-23T01:48:44.4600333Z   inflating: build/bin/cuda_apply_test  
2022-11-23T01:48:44.4658678Z   inflating: build/bin/cuda_caching_host_allocator_test  
2022-11-23T01:48:44.4724392Z   inflating: build/bin/cuda_atomic_ops_test  
2022-11-23T01:48:44.4798635Z   inflating: build/bin/cuda_complex_math_test  
2022-11-23T01:48:44.4853700Z   inflating: build/bin/cuda_device_test  
2022-11-23T01:48:44.4917374Z   inflating: build/bin/cuda_complex_test  
2022-11-23T01:48:44.4983267Z   inflating: build/bin/cuda_cub_test  
2022-11-23T01:48:44.5038345Z   inflating: build/bin/cuda_dlconvertor_test  
2022-11-23T01:48:44.5093063Z   inflating: build/bin/cuda_integer_divider_test  
2022-11-23T01:48:44.5167385Z   inflating: build/bin/cuda_distributions_test  
2022-11-23T01:48:44.5232201Z   inflating: build/bin/cuda_generator_test  
2022-11-23T01:48:44.5286446Z   inflating: build/bin/cuda_half_test  
2022-11-23T01:48:44.5344344Z   inflating: build/bin/cuda_reportMemoryUsage_test  
2022-11-23T01:48:44.5410452Z   inflating: build/bin/cuda_stream_test  
2022-11-23T01:48:44.5464719Z   inflating: build/bin/cuda_optional_test  
2022-11-23T01:48:44.5518089Z   inflating: build/bin/cuda_cudnn_test  
2022-11-23T01:48:44.5574547Z   inflating: build/bin/cuda_packedtensoraccessor_test  
2022-11-23T01:48:44.5631893Z   inflating: build/bin/cuda_vectorized_test  
2022-11-23T01:48:44.5649681Z   inflating: build/bin/tutorial_tensorexpr  
2022-11-23T01:48:44.5720960Z   inflating: build/bin/ProcessGroupGlooTest  
2022-11-23T01:48:44.5784633Z   inflating: build/bin/ProcessGroupGlooAsyncTest  
2022-11-23T01:48:44.5850145Z   inflating: build/bin/ProcessGroupNCCLErrorsTest  
2022-11-23T01:48:44.5917652Z   inflating: build/bin/ProcessGroupNCCLTest  
2022-11-23T01:48:44.5975547Z   inflating: build/bin/ProcessGroupUCCTest  
2022-11-23T01:48:44.6035589Z   inflating: build/bin/test_dist_autograd  
2022-11-23T01:48:44.6112486Z   inflating: build/bin/test_cpp_rpc  
2022-11-23T01:48:44.6113801Z   inflating: build/bin/parallel_benchmark  
2022-11-23T01:48:44.6191725Z   inflating: build/bin/test_mobile_nnc  
2022-11-23T01:48:44.6203988Z   inflating: build/bin/aot_model_compiler_test  
2022-11-23T01:48:44.7141182Z   inflating: build/bin/test_tensorexpr  
2022-11-23T01:48:44.7536664Z   inflating: build/bin/test_lazy     
2022-11-23T01:48:44.7542650Z   inflating: build/bin/torch_shm_manager  
2022-11-23T01:48:44.8903268Z   inflating: build/bin/test_api      
2022-11-23T01:48:45.0149198Z   inflating: build/bin/test_jit      
2022-11-23T01:48:45.0149940Z   inflating: .pytorch-test-times.json  
2022-11-23T01:48:45.0179927Z ##[group]Run df -H
2022-11-23T01:48:45.0180184Z df -H
2022-11-23T01:48:45.0194353Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:48:45.0194650Z env:
2022-11-23T01:48:45.0194892Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:48:45.0195157Z   GPU_FLAG: --gpus all
2022-11-23T01:48:45.0195418Z ##[endgroup]
2022-11-23T01:48:45.0240825Z Filesystem      Size  Used Avail Use% Mounted on
2022-11-23T01:48:45.0241190Z devtmpfs        258G     0  258G   0% /dev
2022-11-23T01:48:45.0241511Z tmpfs           258G     0  258G   0% /dev/shm
2022-11-23T01:48:45.0241770Z tmpfs           258G  750k  258G   1% /run
2022-11-23T01:48:45.0242030Z tmpfs           258G     0  258G   0% /sys/fs/cgroup
2022-11-23T01:48:45.0244963Z /dev/xvda1      162G   30G  132G  19% /
2022-11-23T01:48:45.0245291Z tmpfs            52G     0   52G   0% /run/user/0
2022-11-23T01:48:45.0272043Z ##[group]Run .github/scripts/parse_ref.py
2022-11-23T01:48:45.0272367Z .github/scripts/parse_ref.py
2022-11-23T01:48:45.0286565Z shell: /usr/bin/bash -e {0}
2022-11-23T01:48:45.0286823Z env:
2022-11-23T01:48:45.0287049Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:48:45.0287331Z   GPU_FLAG: --gpus all
2022-11-23T01:48:45.0287592Z ##[endgroup]
2022-11-23T01:48:45.0601269Z ##[group]Run set -x
2022-11-23T01:48:45.0601662Z set -x
2022-11-23T01:48:45.0601909Z 
2022-11-23T01:48:45.0602168Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2022-11-23T01:48:45.0602527Z   TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh
2022-11-23T01:48:45.0602880Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2022-11-23T01:48:45.0603186Z   TEST_COMMAND=.jenkins/caffe2/test.sh
2022-11-23T01:48:45.0603459Z else
2022-11-23T01:48:45.0603895Z   TEST_COMMAND=.jenkins/pytorch/test.sh
2022-11-23T01:48:45.0604163Z fi
2022-11-23T01:48:45.0604358Z 
2022-11-23T01:48:45.0604668Z COMMIT_MESSAGES=$(git cherry -v "origin/${GIT_DEFAULT_BRANCH:-master}")
2022-11-23T01:48:45.0604977Z 
2022-11-23T01:48:45.0605247Z # sanitize the input commit message and PR body here:
2022-11-23T01:48:45.0605532Z #
2022-11-23T01:48:45.0605902Z # trim all new lines from commit messages + PR_BODY to avoid issues with batch environment
2022-11-23T01:48:45.0606500Z # variable copying. see https://github.com/pytorch/pytorch/pull/80043#issuecomment-1167796028
2022-11-23T01:48:45.0606910Z COMMIT_MESSAGES="${COMMIT_MESSAGES//[$'\n\r']}"
2022-11-23T01:48:45.0607215Z PR_BODY="${PR_BODY//[$'\n\r']}"
2022-11-23T01:48:45.0607449Z 
2022-11-23T01:48:45.0607791Z # then trim all special characters like single and double quotes to avoid unescaped inputs to
2022-11-23T01:48:45.0608155Z # wreak havoc internally
2022-11-23T01:48:45.0608469Z export COMMIT_MESSAGES="${COMMIT_MESSAGES//[\'\"]}"
2022-11-23T01:48:45.0608776Z export PR_BODY="${PR_BODY//[\'\"]}"
2022-11-23T01:48:45.0609213Z 
2022-11-23T01:48:45.0609523Z # detached container should get cleaned up by teardown_ec2_linux
2022-11-23T01:48:45.0609911Z # TODO: Stop building test binaries as part of the build phase
2022-11-23T01:48:45.0610286Z # Used for GPU_FLAG since that doesn't play nice
2022-11-23T01:48:45.0610626Z # shellcheck disable=SC2086,SC2090
2022-11-23T01:48:45.0610933Z container_name=$(docker run \
2022-11-23T01:48:45.0611247Z   ${GPU_FLAG:-} \
2022-11-23T01:48:45.0611521Z   -e BUILD_ENVIRONMENT \
2022-11-23T01:48:45.0611793Z   -e PR_NUMBER \
2022-11-23T01:48:45.0612043Z   -e GITHUB_ACTIONS \
2022-11-23T01:48:45.0612303Z   -e BASE_SHA \
2022-11-23T01:48:45.0612551Z   -e BRANCH \
2022-11-23T01:48:45.0612779Z   -e SHA1 \
2022-11-23T01:48:45.0613036Z   -e AWS_DEFAULT_REGION \
2022-11-23T01:48:45.0613306Z   -e IN_WHEEL_TEST \
2022-11-23T01:48:45.0613553Z   -e SHARD_NUMBER \
2022-11-23T01:48:45.0613809Z   -e TEST_CONFIG \
2022-11-23T01:48:45.0614073Z   -e NUM_TEST_SHARDS \
2022-11-23T01:48:45.0614316Z   -e PR_BODY \
2022-11-23T01:48:45.0614584Z   -e COMMIT_MESSAGES \
2022-11-23T01:48:45.0614875Z   -e PYTORCH_RETRY_TEST_CASES \
2022-11-23T01:48:45.0615177Z   -e PYTORCH_OVERRIDE_FLAKY_SIGNAL \
2022-11-23T01:48:45.0615466Z   -e PR_LABELS \
2022-11-23T01:48:45.0615760Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2022-11-23T01:48:45.0616050Z   -e SCCACHE_BUCKET \
2022-11-23T01:48:45.0616312Z   -e SCCACHE_S3_KEY_PREFIX \
2022-11-23T01:48:45.0616583Z   -e XLA_CUDA \
2022-11-23T01:48:45.0617147Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2022-11-23T01:48:45.0617455Z   -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \
2022-11-23T01:48:45.0617791Z   -e PYTORCH_TEST_RERUN_DISABLED_TESTS \
2022-11-23T01:48:45.0618148Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2022-11-23T01:48:45.0618455Z   --ulimit stack=10485760:83886080 \
2022-11-23T01:48:45.0618767Z   --security-opt seccomp=unconfined \
2022-11-23T01:48:45.0619177Z   --cap-add=SYS_PTRACE \
2022-11-23T01:48:45.0619463Z   --ipc=host \
2022-11-23T01:48:45.0619720Z   --shm-size="${SHM_SIZE}" \
2022-11-23T01:48:45.0619983Z   --tty \
2022-11-23T01:48:45.0620374Z   --detach \
2022-11-23T01:48:45.0620840Z   --name="${container_name}" \
2022-11-23T01:48:45.0621110Z   --user jenkins \
2022-11-23T01:48:45.0621421Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2022-11-23T01:48:45.0621740Z   -w /var/lib/jenkins/workspace \
2022-11-23T01:48:45.0622015Z   "${DOCKER_IMAGE}"
2022-11-23T01:48:45.0622247Z )
2022-11-23T01:48:45.0622524Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}"
2022-11-23T01:48:45.0622960Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}"
2022-11-23T01:48:45.0635385Z shell: /usr/bin/bash -e {0}
2022-11-23T01:48:45.0635632Z env:
2022-11-23T01:48:45.0635874Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:48:45.0636123Z   GPU_FLAG: --gpus all
2022-11-23T01:48:45.0636554Z   BUILD_ENVIRONMENT: linux-bionic-cuda11.6-py3.9-gcc7
2022-11-23T01:48:45.0636865Z   PR_NUMBER: 
2022-11-23T01:48:45.0637083Z   BRANCH: master
2022-11-23T01:48:45.0637510Z   SHA1: 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:48:45.0637815Z   BASE_SHA: 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:48:45.0638086Z   PYTORCH_RETRY_TEST_CASES: 1
2022-11-23T01:48:45.0638362Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2022-11-23T01:48:45.0638628Z   TEST_CONFIG: multigpu
2022-11-23T01:48:45.0638866Z   SHARD_NUMBER: 1
2022-11-23T01:48:45.0639080Z   NUM_TEST_SHARDS: 1
2022-11-23T01:48:45.0639307Z   PR_BODY: 
2022-11-23T01:48:45.0639598Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2022-11-23T01:48:45.0639902Z   SCCACHE_S3_KEY_PREFIX: periodic
2022-11-23T01:48:45.0640150Z   SHM_SIZE: 2g
2022-11-23T01:48:45.0640625Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:48:45.0641067Z   XLA_CUDA: 
2022-11-23T01:48:45.0641404Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2022-11-23T01:48:45.0641776Z   PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0
2022-11-23T01:48:45.0642050Z   PYTORCH_TEST_RERUN_DISABLED_TESTS: 1
2022-11-23T01:48:45.0642312Z ##[endgroup]
2022-11-23T01:48:45.0674213Z + [[ multigpu == \m\u\l\t\i\g\p\u ]]
2022-11-23T01:48:45.0674911Z + TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh
2022-11-23T01:48:45.0678585Z ++ git cherry -v origin/master
2022-11-23T01:48:45.0698332Z + COMMIT_MESSAGES=
2022-11-23T01:48:45.0698707Z + COMMIT_MESSAGES=
2022-11-23T01:48:45.0699126Z + PR_BODY=
2022-11-23T01:48:45.0699637Z + export COMMIT_MESSAGES=
2022-11-23T01:48:45.0700182Z + COMMIT_MESSAGES=
2022-11-23T01:48:45.0700788Z + export PR_BODY=
2022-11-23T01:48:45.0701137Z + PR_BODY=
2022-11-23T01:48:45.0713327Z +++ nproc --ignore=2
2022-11-23T01:48:45.0743284Z ++ docker run --gpus all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e COMMIT_MESSAGES -e PYTORCH_RETRY_TEST_CASES -e PYTORCH_OVERRIDE_FLAKY_SIGNAL -e PR_LABELS -e MAX_JOBS=62 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS --env-file=/tmp/github_env_3528394938 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:48:59.3237751Z + container_name=bd7ceb9753ed791b4138e1687453f278a68449d77e2fd64dd747f8de67c8d5c5
2022-11-23T01:48:59.3238349Z + echo DOCKER_CONTAINER_ID=bd7ceb9753ed791b4138e1687453f278a68449d77e2fd64dd747f8de67c8d5c5
2022-11-23T01:48:59.3243956Z ++ echo dist/torch-1.14.0a0+git1cfd385-cp310-cp310-linux_x86_64.whl
2022-11-23T01:48:59.3245198Z + docker exec -t bd7ceb9753ed791b4138e1687453f278a68449d77e2fd64dd747f8de67c8d5c5 sh -c 'pip install dist/torch-1.14.0a0+git1cfd385-cp310-cp310-linux_x86_64.whl[opt-einsum] && .jenkins/pytorch/multigpu-test.sh'
2022-11-23T01:48:59.9252820Z Processing ./dist/torch-1.14.0a0+git1cfd385-cp310-cp310-linux_x86_64.whl
2022-11-23T01:49:00.9182725Z Requirement already satisfied: sympy in /opt/conda/lib/python3.10/site-packages (from torch==1.14.0a0+git1cfd385) (1.11.1)
2022-11-23T01:49:00.9184051Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from torch==1.14.0a0+git1cfd385) (4.4.0)
2022-11-23T01:49:00.9191303Z Requirement already satisfied: networkx in /opt/conda/lib/python3.10/site-packages (from torch==1.14.0a0+git1cfd385) (2.6.3)
2022-11-23T01:49:00.9207139Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/lib/python3.10/site-packages (from torch==1.14.0a0+git1cfd385) (3.3.0)
2022-11-23T01:49:00.9286417Z Requirement already satisfied: numpy>=1.7 in /opt/conda/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==1.14.0a0+git1cfd385) (1.21.2)
2022-11-23T01:49:00.9511613Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.10/site-packages (from sympy->torch==1.14.0a0+git1cfd385) (1.2.1)
2022-11-23T01:49:01.9066956Z Installing collected packages: torch
2022-11-23T01:49:11.9788615Z Successfully installed torch-1.14.0a0+git1cfd385
2022-11-23T01:49:12.0539885Z ++ [[ linux-bionic-cuda11.6-py3.9-gcc7 == *rocm* ]]
2022-11-23T01:49:12.0540239Z ++ BUILD_TEST_LIBTORCH=0
2022-11-23T01:49:12.0540563Z + echo 'Testing pytorch'
2022-11-23T01:49:12.0543412Z Testing pytorch
2022-11-23T01:49:12.0543934Z + python test/run_test.py --verbose -i distributed/test_c10d_common
2022-11-23T01:49:14.4794552Z Ignoring disabled issues:  []
2022-11-23T01:49:14.5323698Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T01:49:14.5324286Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T01:49:14.5324641Z Selected tests:
2022-11-23T01:49:14.5324890Z  distributed/test_c10d_common
2022-11-23T01:49:14.5356290Z Prioritized test from test file changes.
2022-11-23T01:49:14.5356626Z reordering tests for PR:
2022-11-23T01:49:14.5356885Z prioritized: []
2022-11-23T01:49:14.5357378Z the rest: ['distributed/test_c10d_common']
2022-11-23T01:49:14.5357577Z 
2022-11-23T01:49:14.5359844Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T01:49:14.5606502Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T01:49:14.5827906Z parallel (file granularity) tests:
2022-11-23T01:49:14.5828220Z  
2022-11-23T01:49:14.5828486Z serial (file granularity) tests:
2022-11-23T01:49:14.5828754Z  distributed/test_c10d_common
2022-11-23T01:49:16.8451577Z Ignoring disabled issues:  []
2022-11-23T01:49:16.8681887Z Ignoring disabled issues:  []
2022-11-23T01:49:17.2519903Z Running distributed/test_c10d_common ... [2022-11-23 01:49:17.251342]
2022-11-23T01:49:17.2521601Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_common.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 01:49:17.251822]
2022-11-23T01:50:25.6812696Z 
2022-11-23T01:50:25.6813909Z Expand the folded group to see the log file of distributed/test_c10d_common
2022-11-23T01:50:25.6822775Z ##[group]PRINTING LOG FILE of distributed/test_c10d_common (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_common_nbsjmaji)
2022-11-23T01:50:25.6823349Z <unittest.suite.TestSuite tests=[<__main__.CommTest testMethod=test_debug_level>]>
2022-11-23T01:50:25.6823770Z test_debug_level (__main__.CommTest)
2022-11-23T01:50:25.6824553Z <unittest.suite.TestSuite tests=[<__main__.ComputeBucketAssignmentTest testMethod=test_multi_limit_multi_dtype>, <__main__.ComputeBucketAssignmentTest testMethod=test_multi_limit_single_dtype>, <__main__.ComputeBucketAssignmentTest testMethod=test_single_limit_multi_dtype>, <__main__.ComputeBucketAssignmentTest testMethod=test_single_limit_single_dtype>]>
2022-11-23T01:50:25.6825376Z test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest)
2022-11-23T01:50:25.6825821Z test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest)
2022-11-23T01:50:25.6826238Z test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest)
2022-11-23T01:50:25.6826675Z test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest)
2022-11-23T01:50:25.6827535Z <unittest.suite.TestSuite tests=[<__main__.PythonProcessGroupExtensionTest testMethod=test_backend_class_attr>, <__main__.PythonProcessGroupExtensionTest testMethod=test_collectives>, <__main__.PythonProcessGroupExtensionTest testMethod=test_get_backend_name>, <__main__.PythonProcessGroupExtensionTest testMethod=test_send_recv>]>
2022-11-23T01:50:25.6828497Z test_backend_class_attr (__main__.PythonProcessGroupExtensionTest)
2022-11-23T01:50:25.6828916Z test_collectives (__main__.PythonProcessGroupExtensionTest)
2022-11-23T01:50:25.6829423Z test_get_backend_name (__main__.PythonProcessGroupExtensionTest)
2022-11-23T01:50:25.6829853Z test_send_recv (__main__.PythonProcessGroupExtensionTest)
2022-11-23T01:50:25.6830472Z <unittest.suite.TestSuite tests=[<__main__.ReduceOpTest testMethod=test_op_isinstance_of_reduceop>, <__main__.ReduceOpTest testMethod=test_reduceop_copyable>, <__main__.ReduceOpTest testMethod=test_reduceop_pickle>]>
2022-11-23T01:50:25.6831028Z test_op_isinstance_of_reduceop (__main__.ReduceOpTest)
2022-11-23T01:50:25.6831390Z test_reduceop_copyable (__main__.ReduceOpTest)
2022-11-23T01:50:25.6831716Z test_reduceop_pickle (__main__.ReduceOpTest)
2022-11-23T01:50:25.6832432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:50:25.6832866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:50:25.6833440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:50:25.6833916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:50:25.6834155Z 
2022-11-23T01:50:25.6834241Z Running tests...
2022-11-23T01:50:25.6834664Z ----------------------------------------------------------------------
2022-11-23T01:50:25.6835193Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T01:50:25.6835662Z   test_debug_level (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:50:25.6836091Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 370
2022-11-23T01:50:25.6836526Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 371
2022-11-23T01:50:25.6837140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:50:25.6837571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:50:25.6838157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:50:25.6838632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:50:25.6839219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:50:25.6839642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:50:25.6840295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:50:25.6840785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:50:25.6841234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:50:25.6841692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:50:25.6842023Z ok (3.939s)
2022-11-23T01:50:25.6842173Z 
2022-11-23T01:50:25.6842453Z ----------------------------------------------------------------------
2022-11-23T01:50:25.6842765Z Ran 1 test in 3.939s
2022-11-23T01:50:25.6842929Z 
2022-11-23T01:50:25.6843020Z OK
2022-11-23T01:50:25.6843155Z 
2022-11-23T01:50:25.6843282Z Generating XML reports...
2022-11-23T01:50:25.6843833Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20221123014921.xml
2022-11-23T01:50:25.6844637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:50:25.6845090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:50:25.6845761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:50:25.6846213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:50:25.6846444Z 
2022-11-23T01:50:25.6846552Z Running tests...
2022-11-23T01:50:25.6846957Z ----------------------------------------------------------------------
2022-11-23T01:50:25.6847488Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T01:50:25.6847998Z   test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:50:25.6848383Z ok (1.726s)
2022-11-23T01:50:25.6848530Z 
2022-11-23T01:50:25.6848799Z ----------------------------------------------------------------------
2022-11-23T01:50:25.6849105Z Ran 1 test in 1.726s
2022-11-23T01:50:25.6849261Z 
2022-11-23T01:50:25.6849351Z OK
2022-11-23T01:50:25.6849483Z 
2022-11-23T01:50:25.6849611Z Generating XML reports...
2022-11-23T01:50:25.6850239Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221123014927.xml
2022-11-23T01:50:25.6850953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:50:25.6851411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:50:25.6851997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:50:25.6852454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:50:25.6852685Z 
2022-11-23T01:50:25.6852793Z Running tests...
2022-11-23T01:50:25.6853204Z ----------------------------------------------------------------------
2022-11-23T01:50:25.6853745Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T01:50:25.6854259Z   test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:50:25.6854648Z ok (1.729s)
2022-11-23T01:50:25.6854798Z 
2022-11-23T01:50:25.6855070Z ----------------------------------------------------------------------
2022-11-23T01:50:25.6855381Z Ran 1 test in 1.729s
2022-11-23T01:50:25.6855549Z 
2022-11-23T01:50:25.6855648Z OK
2022-11-23T01:50:25.6855787Z 
2022-11-23T01:50:25.6855914Z Generating XML reports...
2022-11-23T01:50:25.6856538Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221123014931.xml
2022-11-23T01:50:25.6857656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:50:25.6858122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:50:25.6858805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:50:25.6859279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:50:25.6859515Z 
2022-11-23T01:50:25.6859626Z Running tests...
2022-11-23T01:50:25.6860042Z ----------------------------------------------------------------------
2022-11-23T01:50:25.6860590Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T01:50:25.6861096Z   test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:50:25.6861489Z ok (1.693s)
2022-11-23T01:50:25.6861639Z 
2022-11-23T01:50:25.6861906Z ----------------------------------------------------------------------
2022-11-23T01:50:25.6862220Z Ran 1 test in 1.693s
2022-11-23T01:50:25.6862386Z 
2022-11-23T01:50:25.6862485Z OK
2022-11-23T01:50:25.6862629Z 
2022-11-23T01:50:25.6862760Z Generating XML reports...
2022-11-23T01:50:25.6863399Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221123014935.xml
2022-11-23T01:50:25.6864195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:50:25.6864649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:50:25.6865230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:50:25.6865681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:50:25.6865912Z 
2022-11-23T01:50:25.6866026Z Running tests...
2022-11-23T01:50:25.6866441Z ----------------------------------------------------------------------
2022-11-23T01:50:25.6866979Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T01:50:25.6867489Z   test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:50:25.6867877Z ok (1.732s)
2022-11-23T01:50:25.6868030Z 
2022-11-23T01:50:25.6868298Z ----------------------------------------------------------------------
2022-11-23T01:50:25.6868619Z Ran 1 test in 1.733s
2022-11-23T01:50:25.6868782Z 
2022-11-23T01:50:25.6868876Z OK
2022-11-23T01:50:25.6869007Z 
2022-11-23T01:50:25.6869130Z Generating XML reports...
2022-11-23T01:50:25.6869748Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221123014939.xml
2022-11-23T01:50:25.6870463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:50:25.6870929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:50:25.6871510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:50:25.6871980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:50:25.6872191Z 
2022-11-23T01:50:25.6872311Z Running tests...
2022-11-23T01:50:25.6872720Z ----------------------------------------------------------------------
2022-11-23T01:50:25.6873261Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T01:50:25.6873782Z   test_backend_class_attr (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:50:25.6874285Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 833
2022-11-23T01:50:25.6874734Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 834
2022-11-23T01:50:25.6875187Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 835
2022-11-23T01:50:25.6875613Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 836
2022-11-23T01:50:25.6876284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:50:25.6876741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:50:25.6877363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:50:25.6877838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:50:25.6878415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:50:25.6878855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:50:25.6879409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:50:25.6879872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:50:25.6880446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:50:25.6880872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:50:25.6881511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:50:25.6881971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:50:25.6882546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:50:25.6882964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:50:25.6883533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:50:25.6883994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:50:25.6884413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:50:25.6884891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:50:25.6885351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:50:25.6885825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:50:25.6886153Z ok (4.143s)
2022-11-23T01:50:25.6886301Z 
2022-11-23T01:50:25.6886572Z ----------------------------------------------------------------------
2022-11-23T01:50:25.6886896Z Ran 1 test in 4.143s
2022-11-23T01:50:25.6887059Z 
2022-11-23T01:50:25.6887133Z OK
2022-11-23T01:50:25.6887265Z 
2022-11-23T01:50:25.6887390Z Generating XML reports...
2022-11-23T01:50:25.6888033Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221123014944.xml
2022-11-23T01:50:25.6888770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:50:25.6889202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:50:25.6889773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:50:25.6890241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:50:25.6890471Z 
2022-11-23T01:50:25.6890581Z Running tests...
2022-11-23T01:50:25.6890965Z ----------------------------------------------------------------------
2022-11-23T01:50:25.6891497Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T01:50:25.6892020Z   test_collectives (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:50:25.6892505Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1164
2022-11-23T01:50:25.6892950Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1165
2022-11-23T01:50:25.6893394Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1166
2022-11-23T01:50:25.6893897Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1167
2022-11-23T01:50:25.6894509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:50:25.6894971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:50:25.6895541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:50:25.6895969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:50:25.6896547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:50:25.6897333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:50:25.6897927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:50:25.6898381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:50:25.6898958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:50:25.6899498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:50:25.6900057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:50:25.6900525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:50:25.6901111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:50:25.6901563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:50:25.6902118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:50:25.6902596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:50:25.6903036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:50:25.6903525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:50:25.6903996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:50:25.6904488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:50:25.6904982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:50:25.6905445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:50:25.6905932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:50:25.6906435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:50:25.6907101Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:50:25.6907788Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:50:25.6908480Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:50:25.6909166Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:50:25.6909553Z ok (5.905s)
2022-11-23T01:50:25.6909682Z 
2022-11-23T01:50:25.6909953Z ----------------------------------------------------------------------
2022-11-23T01:50:25.6910281Z Ran 1 test in 5.905s
2022-11-23T01:50:25.6910443Z 
2022-11-23T01:50:25.6910536Z OK
2022-11-23T01:50:25.6910670Z 
2022-11-23T01:50:25.6910776Z Generating XML reports...
2022-11-23T01:50:25.6911502Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221123014950.xml
2022-11-23T01:50:25.6912260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:50:25.6912773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:50:25.6913337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:50:25.6913805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:50:25.6914035Z 
2022-11-23T01:50:25.6914144Z Running tests...
2022-11-23T01:50:25.6914532Z ----------------------------------------------------------------------
2022-11-23T01:50:25.6915066Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T01:50:25.6915596Z   test_get_backend_name (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:50:25.6916112Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1504
2022-11-23T01:50:25.6916604Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1505
2022-11-23T01:50:25.6917042Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1506
2022-11-23T01:50:25.6917479Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1507
2022-11-23T01:50:25.6918071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:50:25.6918678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:50:25.6919233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:50:25.6919684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:50:25.6920227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:50:25.6920662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:50:25.6921214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:50:25.6921836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:50:25.6922398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:50:25.6922841Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:50:25.6923411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:50:25.6923855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:50:25.6924437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:50:25.6925041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:50:25.6925599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:50:25.6926028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:50:25.6974588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:50:25.6975069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:50:25.6975522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:50:25.6975978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:50:25.6976293Z ok (4.154s)
2022-11-23T01:50:25.6976432Z 
2022-11-23T01:50:25.6977001Z ----------------------------------------------------------------------
2022-11-23T01:50:25.6977518Z Ran 1 test in 4.154s
2022-11-23T01:50:25.6977691Z 
2022-11-23T01:50:25.6977766Z OK
2022-11-23T01:50:25.6977893Z 
2022-11-23T01:50:25.6978008Z Generating XML reports...
2022-11-23T01:50:25.6978676Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221123014958.xml
2022-11-23T01:50:25.6979433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:50:25.6979861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:50:25.6980422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:50:25.6980890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:50:25.6981117Z 
2022-11-23T01:50:25.6981221Z Running tests...
2022-11-23T01:50:25.6981607Z ----------------------------------------------------------------------
2022-11-23T01:50:25.6982133Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T01:50:25.6982740Z   test_send_recv (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:50:25.6983218Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1835
2022-11-23T01:50:25.6983660Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1836
2022-11-23T01:50:25.6984108Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1837
2022-11-23T01:50:25.6984531Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1838
2022-11-23T01:50:25.6985150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:50:25.6985586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:50:25.6986149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:50:25.6986594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:50:25.6987152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:50:25.6987572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:50:25.6988119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:50:25.6988558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:50:25.6989111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:50:25.6989529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:50:25.6990068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:50:25.6990512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:50:25.6991076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:50:25.6991523Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:50:25.6992075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:50:25.6992533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:50:25.6992966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:50:25.6993578Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2022-11-23T01:50:25.6994061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:50:25.6994740Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2022-11-23T01:50:25.6995387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:50:25.6995814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:50:25.6996451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:50:25.6996950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:50:25.6997418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:50:25.6997892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:50:25.6998540Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:50:25.6999224Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:50:25.6999977Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:50:25.7000808Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:50:25.7001191Z ok (6.950s)
2022-11-23T01:50:25.7001328Z 
2022-11-23T01:50:25.7001580Z ----------------------------------------------------------------------
2022-11-23T01:50:25.7001866Z Ran 1 test in 6.950s
2022-11-23T01:50:25.7002013Z 
2022-11-23T01:50:25.7002091Z OK
2022-11-23T01:50:25.7002208Z 
2022-11-23T01:50:25.7002316Z Generating XML reports...
2022-11-23T01:50:25.7003111Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221123015005.xml
2022-11-23T01:50:25.7003838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:50:25.7004276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:50:25.7004843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:50:25.7005304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:50:25.7005679Z 
2022-11-23T01:50:25.7005778Z Running tests...
2022-11-23T01:50:25.7006150Z ----------------------------------------------------------------------
2022-11-23T01:50:25.7006648Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T01:50:25.7007099Z   test_op_isinstance_of_reduceop (__main__.ReduceOpTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:50:25.7007607Z ok (1.676s)
2022-11-23T01:50:25.7007743Z 
2022-11-23T01:50:25.7007997Z ----------------------------------------------------------------------
2022-11-23T01:50:25.7008306Z Ran 1 test in 1.676s
2022-11-23T01:50:25.7008468Z 
2022-11-23T01:50:25.7008563Z OK
2022-11-23T01:50:25.7008695Z 
2022-11-23T01:50:25.7008819Z Generating XML reports...
2022-11-23T01:50:25.7009386Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20221123015015.xml
2022-11-23T01:50:25.7010050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:50:25.7010633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:50:25.7011166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:50:25.7011779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:50:25.7012012Z 
2022-11-23T01:50:25.7012109Z Running tests...
2022-11-23T01:50:25.7012512Z ----------------------------------------------------------------------
2022-11-23T01:50:25.7013176Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T01:50:25.7013644Z   test_reduceop_copyable (__main__.ReduceOpTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:50:25.7013987Z ok (1.724s)
2022-11-23T01:50:25.7014131Z 
2022-11-23T01:50:25.7014385Z ----------------------------------------------------------------------
2022-11-23T01:50:25.7014689Z Ran 1 test in 1.725s
2022-11-23T01:50:25.7014849Z 
2022-11-23T01:50:25.7014942Z OK
2022-11-23T01:50:25.7015074Z 
2022-11-23T01:50:25.7015190Z Generating XML reports...
2022-11-23T01:50:25.7015725Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20221123015019.xml
2022-11-23T01:50:25.7016374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:50:25.7017018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:50:25.7017592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:50:25.7018129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:50:25.7018347Z 
2022-11-23T01:50:25.7018442Z Running tests...
2022-11-23T01:50:25.7018837Z ----------------------------------------------------------------------
2022-11-23T01:50:25.7019367Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T01:50:25.7019827Z   test_reduceop_pickle (__main__.ReduceOpTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:50:25.7020161Z ok (1.740s)
2022-11-23T01:50:25.7020464Z 
2022-11-23T01:50:25.7020719Z ----------------------------------------------------------------------
2022-11-23T01:50:25.7021003Z Ran 1 test in 1.740s
2022-11-23T01:50:25.7021146Z 
2022-11-23T01:50:25.7021222Z OK
2022-11-23T01:50:25.7021343Z 
2022-11-23T01:50:25.7021451Z Generating XML reports...
2022-11-23T01:50:25.7022000Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20221123015023.xml
2022-11-23T01:50:25.7022491Z 
2022-11-23T01:50:25.7022815Z ##[endgroup]
2022-11-23T01:50:25.7023358Z FINISHED PRINTING LOG FILE of distributed/test_c10d_common (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_common_nbsjmaji)
2022-11-23T01:50:25.7023686Z 
2022-11-23T01:50:26.0400321Z 
2022-11-23T01:50:26.0400850Z real	1m13.986s
2022-11-23T01:50:26.0401177Z user	2m17.866s
2022-11-23T01:50:26.0401421Z sys	1m51.159s
2022-11-23T01:50:26.0402008Z + python test/run_test.py --verbose -i distributed/test_c10d_gloo
2022-11-23T01:50:28.3857287Z Ignoring disabled issues:  []
2022-11-23T01:50:28.4368422Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T01:50:28.4369004Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T01:50:28.4369371Z Selected tests:
2022-11-23T01:50:28.4369632Z  distributed/test_c10d_gloo
2022-11-23T01:50:28.4401889Z Prioritized test from test file changes.
2022-11-23T01:50:28.4402427Z reordering tests for PR:
2022-11-23T01:50:28.4402719Z prioritized: []
2022-11-23T01:50:28.4403172Z the rest: ['distributed/test_c10d_gloo']
2022-11-23T01:50:28.4403311Z 
2022-11-23T01:50:28.4403865Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T01:50:28.4404831Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T01:50:28.4409413Z parallel (file granularity) tests:
2022-11-23T01:50:28.4409844Z  
2022-11-23T01:50:28.4410159Z serial (file granularity) tests:
2022-11-23T01:50:28.4410462Z  distributed/test_c10d_gloo
2022-11-23T01:50:30.8014402Z Ignoring disabled issues:  []
2022-11-23T01:50:30.8025083Z Ignoring disabled issues:  []
2022-11-23T01:50:31.2624838Z Running distributed/test_c10d_gloo ... [2022-11-23 01:50:31.261849]
2022-11-23T01:50:31.2626858Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_gloo.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 01:50:31.262294]
2022-11-23T02:05:07.6248254Z 
2022-11-23T02:05:07.6251650Z Expand the folded group to see the log file of distributed/test_c10d_gloo
2022-11-23T02:05:07.6252730Z ##[group]PRINTING LOG FILE of distributed/test_c10d_gloo (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_gloo_yjo3apew)
2022-11-23T02:05:07.6254595Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4_3_k05w
2022-11-23T02:05:07.6255448Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4_3_k05w/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6260542Z <unittest.suite.TestSuite tests=[<__main__.CommTest testMethod=test_broadcast_coalesced_gloo_cpu>, <__main__.CommTest testMethod=test_broadcast_coalesced_gloo_cuda>, <__main__.CommTest testMethod=test_gloo_barrier_device_ids>, <__main__.CommTest testMethod=test_gloo_rank_membership>, <__main__.CommTest testMethod=test_gloo_warn_not_in_group>, <__main__.CommTest testMethod=test_sequence_num_incremented_gloo_default>, <__main__.CommTest testMethod=test_sequence_num_incremented_gloo_subgroup>, <__main__.CommTest testMethod=test_sequence_num_set_default_pg_gloo>, <__main__.CommTest testMethod=test_sequence_num_set_gloo_new_group>, <__main__.CommTest testMethod=test_tensor_dtype_complex>, <__main__.CommTest testMethod=test_tensor_dtype_mismatch>]>
2022-11-23T02:05:07.6263067Z test_broadcast_coalesced_gloo_cpu (__main__.CommTest)
2022-11-23T02:05:07.6263441Z test_broadcast_coalesced_gloo_cuda (__main__.CommTest)
2022-11-23T02:05:07.6263965Z test_gloo_barrier_device_ids (__main__.CommTest)
2022-11-23T02:05:07.6264541Z test_gloo_rank_membership (__main__.CommTest)
2022-11-23T02:05:07.6265145Z test_gloo_warn_not_in_group (__main__.CommTest)
2022-11-23T02:05:07.6265516Z test_sequence_num_incremented_gloo_default (__main__.CommTest)
2022-11-23T02:05:07.6265955Z test_sequence_num_incremented_gloo_subgroup (__main__.CommTest)
2022-11-23T02:05:07.6266372Z test_sequence_num_set_default_pg_gloo (__main__.CommTest)
2022-11-23T02:05:07.6266728Z test_sequence_num_set_gloo_new_group (__main__.CommTest)
2022-11-23T02:05:07.6267050Z test_tensor_dtype_complex (__main__.CommTest)
2022-11-23T02:05:07.6267388Z test_tensor_dtype_mismatch (__main__.CommTest)
2022-11-23T02:05:07.6268663Z <unittest.suite.TestSuite tests=[<__main__.CompilerTest testMethod=test_allgather_work_wait_cpu>, <__main__.CompilerTest testMethod=test_allgather_work_wait_gpu>, <__main__.CompilerTest testMethod=test_allreduce_work_wait_cpu>, <__main__.CompilerTest testMethod=test_allreduce_work_wait_gpu>, <__main__.CompilerTest testMethod=test_broadcast_work_wait_cpu>, <__main__.CompilerTest testMethod=test_broadcast_work_wait_gpu>, <__main__.CompilerTest testMethod=test_consecutive_comm_work_wait_cpu>, <__main__.CompilerTest testMethod=test_consecutive_comm_work_wait_gpu>, <__main__.CompilerTest testMethod=test_nested_comm_tensor_wrapping>, <__main__.CompilerTest testMethod=test_scatter_work_wait_cpu>, <__main__.CompilerTest testMethod=test_scatter_work_wait_gpu>]>
2022-11-23T02:05:07.6270633Z test_allgather_work_wait_cpu (__main__.CompilerTest)
2022-11-23T02:05:07.6271008Z test_allgather_work_wait_gpu (__main__.CompilerTest)
2022-11-23T02:05:07.6271346Z test_allreduce_work_wait_cpu (__main__.CompilerTest)
2022-11-23T02:05:07.6271697Z test_allreduce_work_wait_gpu (__main__.CompilerTest)
2022-11-23T02:05:07.6272052Z test_broadcast_work_wait_cpu (__main__.CompilerTest)
2022-11-23T02:05:07.6272399Z test_broadcast_work_wait_gpu (__main__.CompilerTest)
2022-11-23T02:05:07.6272738Z test_consecutive_comm_work_wait_cpu (__main__.CompilerTest)
2022-11-23T02:05:07.6273114Z test_consecutive_comm_work_wait_gpu (__main__.CompilerTest)
2022-11-23T02:05:07.6273624Z test_nested_comm_tensor_wrapping (__main__.CompilerTest)
2022-11-23T02:05:07.6273987Z test_scatter_work_wait_cpu (__main__.CompilerTest)
2022-11-23T02:05:07.6274338Z test_scatter_work_wait_gpu (__main__.CompilerTest)
2022-11-23T02:05:07.6279891Z <unittest.suite.TestSuite tests=[<__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_module>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_cpu>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_gpu_gloo>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_register_just_once>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_sparse_gradients>, <__main__.DistributedDataParallelTest testMethod=test_ddp_invalid_comm_hook_init>, <__main__.DistributedDataParallelTest testMethod=test_ddp_invalid_comm_hook_return_type>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_when_unused_parameters_empty>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad_with_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad_with_static_graph>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_1gpu_module_device_ids_integer_list>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_1gpu_module_device_ids_torch_device_list>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_2gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_4gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_cpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_cpu_module_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ignored_output>, <__main__.DistributedDataParallelTest testMethod=test_ignored_output_with_unused_parameters>, <__main__.DistributedDataParallelTest testMethod=test_ignored_sharded_tensor>, <__main__.DistributedDataParallelTest testMethod=test_invalid_powerSGD_state>, <__main__.DistributedDataParallelTest testMethod=test_save_load_checkpoint>, <__main__.DistributedDataParallelTest testMethod=test_sparse_gradients>, <__main__.DistributedDataParallelTest testMethod=test_sparse_gradients_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_empty_input>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_only_empty_input>]>
2022-11-23T02:05:07.6285553Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6286047Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6286611Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6287103Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6287624Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6288155Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6288674Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6289148Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6289630Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6290156Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6290675Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6291183Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6291769Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6292557Z test_ddp_comm_hook_future_passing_cpu (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6293158Z test_ddp_comm_hook_future_passing_gpu_gloo (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6293612Z test_ddp_comm_hook_register_just_once (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6294075Z test_ddp_comm_hook_sparse_gradients (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6294532Z test_ddp_invalid_comm_hook_init (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6294979Z test_ddp_invalid_comm_hook_return_type (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6295491Z test_find_unused_parameters_when_unused_parameters_empty (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6295995Z test_global_local_unused_params_grad (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6296498Z test_global_local_unused_params_grad_with_grad_is_view (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6297373Z test_global_local_unused_params_grad_with_static_graph (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6297895Z test_gloo_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6298423Z test_gloo_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6298890Z test_gloo_backend_2gpu_module (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6299479Z test_gloo_backend_4gpu_module (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6299905Z test_gloo_backend_cpu_module (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6300372Z test_gloo_backend_cpu_module_grad_is_view (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6300834Z test_ignored_output (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6301387Z test_ignored_output_with_unused_parameters (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6301833Z test_ignored_sharded_tensor (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6302276Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6302724Z test_save_load_checkpoint (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6303132Z test_sparse_gradients (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6303583Z test_sparse_gradients_grad_is_view (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6304041Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6304503Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6305718Z <unittest.suite.TestSuite tests=[<__main__.GlooProcessGroupWithDispatchedCollectivesTests testMethod=test_allgather_coalesced>, <__main__.GlooProcessGroupWithDispatchedCollectivesTests testMethod=test_allreduce_coalesced>, <__main__.GlooProcessGroupWithDispatchedCollectivesTests testMethod=test_collectives>, <__main__.GlooProcessGroupWithDispatchedCollectivesTests testMethod=test_monitored_barrier>]>
2022-11-23T02:05:07.6306895Z test_allgather_coalesced (__main__.GlooProcessGroupWithDispatchedCollectivesTests)
2022-11-23T02:05:07.6307482Z test_allreduce_coalesced (__main__.GlooProcessGroupWithDispatchedCollectivesTests)
2022-11-23T02:05:07.6308046Z test_collectives (__main__.GlooProcessGroupWithDispatchedCollectivesTests)
2022-11-23T02:05:07.6308609Z test_monitored_barrier (__main__.GlooProcessGroupWithDispatchedCollectivesTests)
2022-11-23T02:05:07.6309029Z <unittest.suite.TestSuite tests=[]>
2022-11-23T02:05:07.6314273Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupGlooTest testMethod=test_allgather_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_coalesced_async>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_coalesced_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_noncontiguous_input>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_cuda_using_work_api>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_using_work_api>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_async>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_checks_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_barrier_implies_wait>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_basics>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_checks>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_stress>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_empty_tensors>, <__main__.ProcessGroupGlooTest testMethod=test_gather_basics>, <__main__.ProcessGroupGlooTest testMethod=test_gather_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_gather_checks>, <__main__.ProcessGroupGlooTest testMethod=test_gather_noncontiguous_input>, <__main__.ProcessGroupGlooTest testMethod=test_gather_stress>, <__main__.ProcessGroupGlooTest testMethod=test_gather_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_multi_device_constructor>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_checks>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_stress>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_round_robin>, <__main__.ProcessGroupGlooTest testMethod=test_round_robin_create_destroy>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_basics>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_checks>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_stress>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_send_recv_all_to_all>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_checks>]>
2022-11-23T02:05:07.6319506Z test_allgather_basics (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6319924Z test_allgather_basics_cuda (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6320322Z test_allgather_checks (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6320705Z test_allgather_coalesced_async (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6321127Z test_allgather_coalesced_checks (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6321563Z test_allgather_noncontiguous_input (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6321948Z test_allgather_stress (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6322351Z test_allgather_stress_cuda (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6322748Z test_allreduce_basics (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6323117Z test_allreduce_basics_cuda (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6323605Z test_allreduce_basics_cuda_using_work_api (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6324049Z test_allreduce_basics_using_work_api (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6324466Z test_allreduce_checks (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6324839Z test_allreduce_coalesced_async (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6325267Z test_allreduce_coalesced_basics (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6325682Z test_allreduce_coalesced_checks (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6326086Z test_allreduce_coalesced_checks_cuda (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6326517Z test_allreduce_coalesced_stress (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6326919Z test_allreduce_stress (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6327315Z test_allreduce_stress_cuda (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6327696Z test_barrier_implies_wait (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6328095Z test_broadcast_basics (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6328487Z test_broadcast_basics_cuda (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6328860Z test_broadcast_checks (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6329237Z test_broadcast_stress (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6329630Z test_broadcast_stress_cuda (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6329984Z test_empty_tensors (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6330360Z test_gather_basics (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6330744Z test_gather_basics_cuda (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6331099Z test_gather_checks (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6331509Z test_gather_noncontiguous_input (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6331907Z test_gather_stress (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6332286Z test_gather_stress_cuda (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6332672Z test_multi_device_constructor (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6333060Z test_reduce_basics (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6333448Z test_reduce_basics_cuda (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6333801Z test_reduce_checks (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6334173Z test_reduce_stress (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6334549Z test_reduce_stress_cuda (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6334906Z test_round_robin (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6335298Z test_round_robin_create_destroy (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6335698Z test_scatter_basics (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6336083Z test_scatter_basics_cuda (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6336446Z test_scatter_checks (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6337237Z test_scatter_stress (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6337659Z test_scatter_stress_cuda (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6338040Z test_send_recv_all_to_all (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6338448Z test_sparse_allreduce_basics (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6338858Z test_sparse_allreduce_basics_cuda (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6339254Z test_sparse_allreduce_checks (__main__.ProcessGroupGlooTest)
2022-11-23T02:05:07.6340235Z <unittest.suite.TestSuite tests=[<__main__.ReducerTest testMethod=test_forward_backward>, <__main__.ReducerTest testMethod=test_forward_backward_optimizer>, <__main__.ReducerTest testMethod=test_forward_backward_unused_parameters>, <__main__.ReducerTest testMethod=test_multi_dtype_multi_bucket>, <__main__.ReducerTest testMethod=test_multi_dtype_single_bucket>, <__main__.ReducerTest testMethod=test_single_dtype_single_bucket>]>
2022-11-23T02:05:07.6341070Z test_forward_backward (__main__.ReducerTest)
2022-11-23T02:05:07.6341436Z test_forward_backward_optimizer (__main__.ReducerTest)
2022-11-23T02:05:07.6341914Z test_forward_backward_unused_parameters (__main__.ReducerTest)
2022-11-23T02:05:07.6342266Z test_multi_dtype_multi_bucket (__main__.ReducerTest)
2022-11-23T02:05:07.6342628Z test_multi_dtype_single_bucket (__main__.ReducerTest)
2022-11-23T02:05:07.6343000Z test_single_dtype_single_bucket (__main__.ReducerTest)
2022-11-23T02:05:07.6343418Z <unittest.suite.TestSuite tests=[<__main__.RendezvousEnvTest testMethod=test_logging_init>]>
2022-11-23T02:05:07.6343847Z test_logging_init (__main__.RendezvousEnvTest)
2022-11-23T02:05:07.6344182Z <unittest.suite.TestSuite tests=[]>
2022-11-23T02:05:07.6344603Z <unittest.suite.TestSuite tests=[<__main__.TimeoutTest testMethod=test_default_store_timeout_gloo>]>
2022-11-23T02:05:07.6345046Z test_default_store_timeout_gloo (__main__.TimeoutTest)
2022-11-23T02:05:07.6345776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6346256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6346840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6347336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6347828Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6e2gvfab
2022-11-23T02:05:07.6348365Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6e2gvfab/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6348693Z 
2022-11-23T02:05:07.6348811Z Running tests...
2022-11-23T02:05:07.6349244Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6349804Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6350295Z   test_broadcast_coalesced_gloo_cpu (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6350787Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2653
2022-11-23T02:05:07.6351261Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2654
2022-11-23T02:05:07.6351894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6352351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6352957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6353456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6354028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6354502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6355170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6355683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6356143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqo_1h166
2022-11-23T02:05:07.6356699Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqo_1h166/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6357227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6357740Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd1f_cp7z
2022-11-23T02:05:07.6358274Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd1f_cp7z/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6358802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6359158Z ok (4.144s)
2022-11-23T02:05:07.6359311Z 
2022-11-23T02:05:07.6359578Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6360003Z Ran 1 test in 4.144s
2022-11-23T02:05:07.6360176Z 
2022-11-23T02:05:07.6360276Z OK
2022-11-23T02:05:07.6360421Z 
2022-11-23T02:05:07.6360556Z Generating XML reports...
2022-11-23T02:05:07.6361108Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123015035.xml
2022-11-23T02:05:07.6361792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6362265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6362832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6363330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6363819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe1kxl644
2022-11-23T02:05:07.6364384Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe1kxl644/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6364678Z 
2022-11-23T02:05:07.6364791Z Running tests...
2022-11-23T02:05:07.6365212Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6365901Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6366389Z   test_broadcast_coalesced_gloo_cuda (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6366876Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2858
2022-11-23T02:05:07.6367340Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2859
2022-11-23T02:05:07.6367972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6368421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6369020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6369521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6370122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6370568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6371151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6371634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6372100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzbm07b7x
2022-11-23T02:05:07.6372670Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzbm07b7x/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6373293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp6awvcmg
2022-11-23T02:05:07.6373872Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp6awvcmg/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6374383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6374867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6375227Z ok (5.857s)
2022-11-23T02:05:07.6375381Z 
2022-11-23T02:05:07.6375644Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6375991Z Ran 1 test in 5.857s
2022-11-23T02:05:07.6376164Z 
2022-11-23T02:05:07.6376266Z OK
2022-11-23T02:05:07.6376406Z 
2022-11-23T02:05:07.6376573Z Generating XML reports...
2022-11-23T02:05:07.6377466Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123015041.xml
2022-11-23T02:05:07.6378158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6378626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6379306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6379804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6380291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp2jgv9yz
2022-11-23T02:05:07.6380860Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp2jgv9yz/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6381171Z 
2022-11-23T02:05:07.6381268Z Running tests...
2022-11-23T02:05:07.6381688Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6382242Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6382723Z   test_gloo_barrier_device_ids (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6383205Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3065
2022-11-23T02:05:07.6383670Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3066
2022-11-23T02:05:07.6384332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6386621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6387238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6387723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6388324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6388767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6389365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6389866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6390331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpafk6v6_o
2022-11-23T02:05:07.6390899Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpafk6v6_o/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6391450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1l3ugwal
2022-11-23T02:05:07.6392010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1l3ugwal/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6392524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6393018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6393594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:05:07.6394102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:05:07.6394801Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6395516Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6395925Z ok (4.117s)
2022-11-23T02:05:07.6396084Z 
2022-11-23T02:05:07.6396334Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6396675Z Ran 1 test in 4.117s
2022-11-23T02:05:07.6396845Z 
2022-11-23T02:05:07.6396942Z OK
2022-11-23T02:05:07.6397078Z 
2022-11-23T02:05:07.6397210Z Generating XML reports...
2022-11-23T02:05:07.6397751Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123015050.xml
2022-11-23T02:05:07.6398443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6398981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6399555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6400054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6400543Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd876xoth
2022-11-23T02:05:07.6401102Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd876xoth/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6401420Z 
2022-11-23T02:05:07.6401512Z Running tests...
2022-11-23T02:05:07.6401937Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6402477Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6402954Z   test_gloo_rank_membership (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6403430Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3270
2022-11-23T02:05:07.6403896Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3271
2022-11-23T02:05:07.6404523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6404962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6405549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6406034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6406633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6407083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6407675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6408160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6408612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpunro014g
2022-11-23T02:05:07.6409174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpunro014g/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6409726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5s3x4mhe
2022-11-23T02:05:07.6410243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6410750Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5s3x4mhe/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6411335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6411857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:05:07.6412555Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6413088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:05:07.6413766Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6414316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:05:07.6414831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:05:07.6415475Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:05:07.6416189Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:05:07.6416659Z ok (4.093s)
2022-11-23T02:05:07.6417126Z 
2022-11-23T02:05:07.6417392Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6417744Z Ran 1 test in 4.093s
2022-11-23T02:05:07.6417909Z 
2022-11-23T02:05:07.6418009Z OK
2022-11-23T02:05:07.6418154Z 
2022-11-23T02:05:07.6418291Z Generating XML reports...
2022-11-23T02:05:07.6418826Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123015056.xml
2022-11-23T02:05:07.6419514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6419985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6420551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6421053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6421534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbnb52ya1
2022-11-23T02:05:07.6422092Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbnb52ya1/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6422402Z 
2022-11-23T02:05:07.6422493Z Running tests...
2022-11-23T02:05:07.6422918Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6423463Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6423958Z   test_gloo_warn_not_in_group (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6424411Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3478
2022-11-23T02:05:07.6424873Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3479
2022-11-23T02:05:07.6425506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6425956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6426546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6427035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6427639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6428081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6428683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6429169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6429717Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps9tbm3tm
2022-11-23T02:05:07.6430293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps9tbm3tm/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6430826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6431344Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcgrqlgb2
2022-11-23T02:05:07.6431874Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcgrqlgb2/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6432394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6432893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:05:07.6433399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:05:07.6434067Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6434777Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6435410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:05:07.6435919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:05:07.6436561Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:05:07.6437255Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:05:07.6437656Z ok (5.771s)
2022-11-23T02:05:07.6437808Z 
2022-11-23T02:05:07.6438056Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6438383Z Ran 1 test in 5.771s
2022-11-23T02:05:07.6438547Z 
2022-11-23T02:05:07.6438648Z OK
2022-11-23T02:05:07.6438788Z 
2022-11-23T02:05:07.6438918Z Generating XML reports...
2022-11-23T02:05:07.6439459Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123015103.xml
2022-11-23T02:05:07.6440256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6440729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6441301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6441795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6442275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpynidyv0l
2022-11-23T02:05:07.6442829Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpynidyv0l/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6443113Z 
2022-11-23T02:05:07.6443228Z Running tests...
2022-11-23T02:05:07.6443646Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6444186Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6444674Z   test_sequence_num_incremented_gloo_default (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6445169Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3688
2022-11-23T02:05:07.6445637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3689
2022-11-23T02:05:07.6446275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6446723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6447318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6447856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6448442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6448909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6449501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6449967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6450415Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjxm_9ytv
2022-11-23T02:05:07.6450956Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjxm_9ytv/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6451471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6451980Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzmp124nx
2022-11-23T02:05:07.6452507Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzmp124nx/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6453080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6453570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:05:07.6454047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:05:07.6454712Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6455410Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6455946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:05:07.6456425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:05:07.6457302Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:05:07.6458008Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:05:07.6458405Z ok (5.848s)
2022-11-23T02:05:07.6458539Z 
2022-11-23T02:05:07.6458809Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6459142Z Ran 1 test in 5.848s
2022-11-23T02:05:07.6459304Z 
2022-11-23T02:05:07.6459395Z OK
2022-11-23T02:05:07.6459529Z 
2022-11-23T02:05:07.6459635Z Generating XML reports...
2022-11-23T02:05:07.6460180Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123015111.xml
2022-11-23T02:05:07.6460850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6461303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6461863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6462332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6462799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnw8gv67s
2022-11-23T02:05:07.6463328Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnw8gv67s/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6463634Z 
2022-11-23T02:05:07.6463742Z Running tests...
2022-11-23T02:05:07.6464155Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6464682Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6465168Z   test_sequence_num_incremented_gloo_subgroup (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6465736Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3901
2022-11-23T02:05:07.6466202Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3902
2022-11-23T02:05:07.6466798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6467252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6467829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6468301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6468861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6469309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6469886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6470349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6470872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpidwt_j1a
2022-11-23T02:05:07.6471420Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpidwt_j1a/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6471956Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3dmmm7qf
2022-11-23T02:05:07.6472556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3dmmm7qf/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6473079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6473552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6473900Z ok (4.035s)
2022-11-23T02:05:07.6474030Z 
2022-11-23T02:05:07.6474315Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6474648Z Ran 1 test in 4.035s
2022-11-23T02:05:07.6474814Z 
2022-11-23T02:05:07.6474909Z OK
2022-11-23T02:05:07.6475047Z 
2022-11-23T02:05:07.6475153Z Generating XML reports...
2022-11-23T02:05:07.6475702Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123015119.xml
2022-11-23T02:05:07.6476362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6476815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6477374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6477844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6478309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb5zkosf3
2022-11-23T02:05:07.6478841Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb5zkosf3/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6479153Z 
2022-11-23T02:05:07.6479263Z Running tests...
2022-11-23T02:05:07.6479666Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6480198Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6480671Z   test_sequence_num_set_default_pg_gloo (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6481138Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4100
2022-11-23T02:05:07.6481585Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4101
2022-11-23T02:05:07.6482177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6482624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6483359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6483852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6484417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6484867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6485437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6485904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6486354Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjja3tfws
2022-11-23T02:05:07.6486901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjja3tfws/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6487413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6487895Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7d570lwx
2022-11-23T02:05:07.6488506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7d570lwx/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6489019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6489509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:05:07.6489985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:05:07.6490651Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6491349Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6491746Z ok (4.161s)
2022-11-23T02:05:07.6491882Z 
2022-11-23T02:05:07.6492150Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6492483Z Ran 1 test in 4.161s
2022-11-23T02:05:07.6492647Z 
2022-11-23T02:05:07.6492741Z OK
2022-11-23T02:05:07.6492859Z 
2022-11-23T02:05:07.6492986Z Generating XML reports...
2022-11-23T02:05:07.6493537Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123015126.xml
2022-11-23T02:05:07.6494202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6494633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6495210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6495679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6496150Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1pljao_7
2022-11-23T02:05:07.6496680Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1pljao_7/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6497228Z 
2022-11-23T02:05:07.6497341Z Running tests...
2022-11-23T02:05:07.6497764Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6498304Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6498773Z   test_sequence_num_set_gloo_new_group (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6499241Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4305
2022-11-23T02:05:07.6499689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4306
2022-11-23T02:05:07.6500283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6500820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6501424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6501901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6502469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6502913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6503487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6503933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6504403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5znt2nsr
2022-11-23T02:05:07.6504942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5znt2nsr/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6505459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6506026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwetvc_rm
2022-11-23T02:05:07.6506564Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwetvc_rm/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6507074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6507569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:05:07.6508050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:05:07.6508725Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6509428Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6509945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:05:07.6510442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:05:07.6511099Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:05:07.6511789Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:05:07.6512164Z ok (4.209s)
2022-11-23T02:05:07.6512313Z 
2022-11-23T02:05:07.6512584Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6512912Z Ran 1 test in 4.209s
2022-11-23T02:05:07.6513072Z 
2022-11-23T02:05:07.6513164Z OK
2022-11-23T02:05:07.6513279Z 
2022-11-23T02:05:07.6513402Z Generating XML reports...
2022-11-23T02:05:07.6513953Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123015132.xml
2022-11-23T02:05:07.6514629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6515062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6515643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6516113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6516583Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsjvyr838
2022-11-23T02:05:07.6517113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsjvyr838/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6517415Z 
2022-11-23T02:05:07.6517525Z Running tests...
2022-11-23T02:05:07.6517930Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6518501Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6518994Z   test_tensor_dtype_complex (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6519457Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4516
2022-11-23T02:05:07.6519903Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4517
2022-11-23T02:05:07.6520497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6520946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6521522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6521974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6522555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6522999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6523632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6524077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6524548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvbie5p07
2022-11-23T02:05:07.6525096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvbie5p07/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6525588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6526089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_yf6owv8
2022-11-23T02:05:07.6526623Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_yf6owv8/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6527136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6527613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:05:07.6528280Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6528817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:05:07.6529475Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6529849Z ok (4.153s)
2022-11-23T02:05:07.6529995Z 
2022-11-23T02:05:07.6530265Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6530594Z Ran 1 test in 4.154s
2022-11-23T02:05:07.6530753Z 
2022-11-23T02:05:07.6530827Z OK
2022-11-23T02:05:07.6530958Z 
2022-11-23T02:05:07.6531082Z Generating XML reports...
2022-11-23T02:05:07.6531632Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123015139.xml
2022-11-23T02:05:07.6532304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6532737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6533312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6533780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6534227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3fzh3g5n
2022-11-23T02:05:07.6534771Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3fzh3g5n/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6535076Z 
2022-11-23T02:05:07.6535182Z Running tests...
2022-11-23T02:05:07.6535653Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6536183Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6536658Z   test_tensor_dtype_mismatch (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6537376Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4721
2022-11-23T02:05:07.6537812Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4722
2022-11-23T02:05:07.6538437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6538967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6539556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6540101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6540697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6541249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6541835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6542290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6542759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6j9r1h15
2022-11-23T02:05:07.6543302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6j9r1h15/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6543826Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpulvcbgfe
2022-11-23T02:05:07.6544370Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpulvcbgfe/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6544894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6545371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6545848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:05:07.6546344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:05:07.6547011Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6547707Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6548754Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:05:07.6549401Z   warnings.warn(
2022-11-23T02:05:07.6550281Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:05:07.6550899Z   warnings.warn(
2022-11-23T02:05:07.6551748Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:05:07.6552369Z   warnings.warn(
2022-11-23T02:05:07.6553309Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:05:07.6553947Z   warnings.warn(
2022-11-23T02:05:07.6554167Z ok (4.116s)
2022-11-23T02:05:07.6554314Z 
2022-11-23T02:05:07.6554591Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6554921Z Ran 1 test in 4.117s
2022-11-23T02:05:07.6555089Z 
2022-11-23T02:05:07.6555185Z OK
2022-11-23T02:05:07.6555298Z 
2022-11-23T02:05:07.6555423Z Generating XML reports...
2022-11-23T02:05:07.6555970Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123015146.xml
2022-11-23T02:05:07.6556636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6557066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6557643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6558124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6558658Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyharxgax
2022-11-23T02:05:07.6559190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyharxgax/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6559497Z 
2022-11-23T02:05:07.6559610Z Running tests...
2022-11-23T02:05:07.6560027Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6560543Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6561035Z   test_allgather_work_wait_cpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6561505Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4926
2022-11-23T02:05:07.6561954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4927
2022-11-23T02:05:07.6562549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6563001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6563580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6564034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6564620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6565061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6565637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6566083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6566556Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnkg0qxr5
2022-11-23T02:05:07.6567107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnkg0qxr5/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6567633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz2rcb4q9
2022-11-23T02:05:07.6568174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz2rcb4q9/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6568688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6569166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6569638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:05:07.6570133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:05:07.6570855Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6571562Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6572480Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6573212Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6574075Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6574794Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6575105Z ok (4.160s)
2022-11-23T02:05:07.6575255Z 
2022-11-23T02:05:07.6575523Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6575912Z Ran 1 test in 4.160s
2022-11-23T02:05:07.6576079Z 
2022-11-23T02:05:07.6576174Z OK
2022-11-23T02:05:07.6576289Z 
2022-11-23T02:05:07.6576413Z Generating XML reports...
2022-11-23T02:05:07.6577408Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123015152.xml
2022-11-23T02:05:07.6578101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6578535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6579116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6579590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6580068Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphp__hw_3
2022-11-23T02:05:07.6580593Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphp__hw_3/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6580900Z 
2022-11-23T02:05:07.6581010Z Running tests...
2022-11-23T02:05:07.6581417Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6581931Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6582420Z   test_allgather_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6582895Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5131
2022-11-23T02:05:07.6583344Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5132
2022-11-23T02:05:07.6583936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6584392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6584971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6585431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6586015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6586465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6587040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6587489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6587958Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpey_fendq
2022-11-23T02:05:07.6588584Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpey_fendq/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6589116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6589605Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpal8yk836
2022-11-23T02:05:07.6590143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpal8yk836/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6590658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6591128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:05:07.6591635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:05:07.6592309Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6593012Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6593926Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6594739Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6595601Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6596334Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6596645Z ok (5.876s)
2022-11-23T02:05:07.6596794Z 
2022-11-23T02:05:07.6597065Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6597395Z Ran 1 test in 5.876s
2022-11-23T02:05:07.6597555Z 
2022-11-23T02:05:07.6597644Z OK
2022-11-23T02:05:07.6597765Z 
2022-11-23T02:05:07.6597886Z Generating XML reports...
2022-11-23T02:05:07.6598446Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123015159.xml
2022-11-23T02:05:07.6599122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6599552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6600129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6600597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6601141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4l5bdod_
2022-11-23T02:05:07.6601679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4l5bdod_/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6601983Z 
2022-11-23T02:05:07.6602094Z Running tests...
2022-11-23T02:05:07.6602515Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6603028Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6603514Z   test_allreduce_work_wait_cpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6603984Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5338
2022-11-23T02:05:07.6604429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5339
2022-11-23T02:05:07.6605025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6605471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6606119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6606583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6607166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6607611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6608180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6608626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6609097Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1tvy23gz
2022-11-23T02:05:07.6609640Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1tvy23gz/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6610155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp28i0akbm
2022-11-23T02:05:07.6610691Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp28i0akbm/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6611268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6611740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6612207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:05:07.6612697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:05:07.6613361Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6614052Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6614958Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6615688Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6616539Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6617539Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6618385Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6619103Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6619943Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6620651Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6620973Z ok (4.141s)
2022-11-23T02:05:07.6621103Z 
2022-11-23T02:05:07.6621370Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6621696Z Ran 1 test in 4.141s
2022-11-23T02:05:07.6621853Z 
2022-11-23T02:05:07.6621943Z OK
2022-11-23T02:05:07.6622077Z 
2022-11-23T02:05:07.6622183Z Generating XML reports...
2022-11-23T02:05:07.6622740Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123015207.xml
2022-11-23T02:05:07.6623538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6624007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6624580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6625051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6625521Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0vtvy5fk
2022-11-23T02:05:07.6626049Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0vtvy5fk/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6626352Z 
2022-11-23T02:05:07.6626458Z Running tests...
2022-11-23T02:05:07.6626860Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6627386Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6627855Z   test_allreduce_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6628322Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5543
2022-11-23T02:05:07.6628850Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5544
2022-11-23T02:05:07.6629444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6629898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6630461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6630903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6631462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6631929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6632515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6632965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6633427Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyf3naicy
2022-11-23T02:05:07.6633973Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyf3naicy/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6634507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6j7njwlu
2022-11-23T02:05:07.6635028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6j7njwlu/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6635534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6636003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6636493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:05:07.6636973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:05:07.6637635Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6638329Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6639256Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6640051Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6640970Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6641693Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6642530Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6643219Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6644054Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6644759Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6645084Z ok (5.867s)
2022-11-23T02:05:07.6645289Z 
2022-11-23T02:05:07.6645541Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6645869Z Ran 1 test in 5.867s
2022-11-23T02:05:07.6646032Z 
2022-11-23T02:05:07.6646125Z OK
2022-11-23T02:05:07.6646258Z 
2022-11-23T02:05:07.6646383Z Generating XML reports...
2022-11-23T02:05:07.6646924Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123015214.xml
2022-11-23T02:05:07.6647599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6648045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6648601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6649069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6649531Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpytbd72w0
2022-11-23T02:05:07.6650078Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpytbd72w0/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6650362Z 
2022-11-23T02:05:07.6650468Z Running tests...
2022-11-23T02:05:07.6650871Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6651396Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6651862Z   test_broadcast_work_wait_cpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6652331Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5750
2022-11-23T02:05:07.6652777Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5751
2022-11-23T02:05:07.6653388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6653821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6654403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6654869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6655425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6655867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6656438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6657176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6657629Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5hvg7zar
2022-11-23T02:05:07.6658252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5hvg7zar/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6658805Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdw3x1kux
2022-11-23T02:05:07.6659346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdw3x1kux/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6659834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6660306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6660792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:05:07.6661450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6661985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:05:07.6662641Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6663646Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6664349Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6665203Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6665914Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6666240Z ok (4.126s)
2022-11-23T02:05:07.6666388Z 
2022-11-23T02:05:07.6666640Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6666968Z Ran 1 test in 4.127s
2022-11-23T02:05:07.6667130Z 
2022-11-23T02:05:07.6667222Z OK
2022-11-23T02:05:07.6667355Z 
2022-11-23T02:05:07.6667479Z Generating XML reports...
2022-11-23T02:05:07.6668026Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123015222.xml
2022-11-23T02:05:07.6668702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6669152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6669711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6670185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6670648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaxoq3l73
2022-11-23T02:05:07.6671189Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaxoq3l73/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6671496Z 
2022-11-23T02:05:07.6671586Z Running tests...
2022-11-23T02:05:07.6671986Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6672515Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6672979Z   test_broadcast_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6673450Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5955
2022-11-23T02:05:07.6673892Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5956
2022-11-23T02:05:07.6674498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6674934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6675568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6676054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6676632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6677057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6677631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6678096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6678553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv3qkop12
2022-11-23T02:05:07.6679097Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv3qkop12/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6679633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmoz34ffj
2022-11-23T02:05:07.6680170Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmoz34ffj/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6680728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6681199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6681684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:05:07.6682164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:05:07.6682832Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6683521Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6684452Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6685175Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6686012Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6686723Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6687049Z ok (5.748s)
2022-11-23T02:05:07.6687194Z 
2022-11-23T02:05:07.6687443Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6687765Z Ran 1 test in 5.749s
2022-11-23T02:05:07.6687927Z 
2022-11-23T02:05:07.6688023Z OK
2022-11-23T02:05:07.6688155Z 
2022-11-23T02:05:07.6688279Z Generating XML reports...
2022-11-23T02:05:07.6688830Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123015229.xml
2022-11-23T02:05:07.6689508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6689958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6690516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6690986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6691455Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpilleqto5
2022-11-23T02:05:07.6691999Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpilleqto5/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6692300Z 
2022-11-23T02:05:07.6692452Z Running tests...
2022-11-23T02:05:07.6692869Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6693399Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6693898Z   test_consecutive_comm_work_wait_cpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6694356Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6162
2022-11-23T02:05:07.6694800Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6163
2022-11-23T02:05:07.6695410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6695845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6696418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6697166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6697755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6698278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6698855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6699316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6699764Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjnfx_g2o
2022-11-23T02:05:07.6700306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjnfx_g2o/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6700824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6701328Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpppn1szuq
2022-11-23T02:05:07.6701851Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpppn1szuq/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6702370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6702857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:05:07.6703353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:05:07.6703999Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6704686Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6705611Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6706334Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6707164Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6707880Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6708719Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6709421Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6710311Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6711033Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6711869Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6712579Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6713423Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6714111Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6715025Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6715735Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6716572Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6717262Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6717580Z ok (4.225s)
2022-11-23T02:05:07.6717729Z 
2022-11-23T02:05:07.6718001Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6718322Z Ran 1 test in 4.225s
2022-11-23T02:05:07.6718469Z 
2022-11-23T02:05:07.6718558Z OK
2022-11-23T02:05:07.6718689Z 
2022-11-23T02:05:07.6718810Z Generating XML reports...
2022-11-23T02:05:07.6719370Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123015237.xml
2022-11-23T02:05:07.6720028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6720480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6721058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6721533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6721983Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5vth0l2y
2022-11-23T02:05:07.6722533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5vth0l2y/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6722836Z 
2022-11-23T02:05:07.6722944Z Running tests...
2022-11-23T02:05:07.6723334Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6723860Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6724351Z   test_consecutive_comm_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6724826Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6367
2022-11-23T02:05:07.6725254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6368
2022-11-23T02:05:07.6725859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6726305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6726936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6727421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6728001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6728442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6729000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6729463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6729929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpea6pzoao
2022-11-23T02:05:07.6730478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpea6pzoao/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6730975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6731480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq2rv5mgh
2022-11-23T02:05:07.6732088Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq2rv5mgh/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6732579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6733067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:05:07.6733571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:05:07.6734243Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6734919Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6735848Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6736570Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6737712Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6738421Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6739272Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6740198Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6741063Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6741771Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6742599Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6743313Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6744239Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6744976Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6745805Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6746528Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6747384Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6748096Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6748427Z ok (5.755s)
2022-11-23T02:05:07.6748675Z 
2022-11-23T02:05:07.6748953Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6749295Z Ran 1 test in 5.755s
2022-11-23T02:05:07.6749462Z 
2022-11-23T02:05:07.6749559Z OK
2022-11-23T02:05:07.6749699Z 
2022-11-23T02:05:07.6749805Z Generating XML reports...
2022-11-23T02:05:07.6750380Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123015244.xml
2022-11-23T02:05:07.6751057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6751526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6752090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6752562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6753036Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9v7agauf
2022-11-23T02:05:07.6753572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9v7agauf/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6753875Z 
2022-11-23T02:05:07.6753986Z Running tests...
2022-11-23T02:05:07.6754398Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6754938Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6755418Z   test_nested_comm_tensor_wrapping (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6755896Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6574
2022-11-23T02:05:07.6790287Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6575
2022-11-23T02:05:07.6791071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6791519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6792114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6792590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6793173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6793601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6794180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6794644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6795097Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyfvsuhut
2022-11-23T02:05:07.6795827Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyfvsuhut/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6796372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6796874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_2y_3w6b
2022-11-23T02:05:07.6797393Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_2y_3w6b/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6797898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6798388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:05:07.6798874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:05:07.6799554Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6800255Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6801287Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6802021Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6802855Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6803570Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6804427Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6805147Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6805980Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6806695Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6807021Z ok (4.166s)
2022-11-23T02:05:07.6807171Z 
2022-11-23T02:05:07.6807441Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6807753Z Ran 1 test in 4.166s
2022-11-23T02:05:07.6807915Z 
2022-11-23T02:05:07.6808007Z OK
2022-11-23T02:05:07.6808141Z 
2022-11-23T02:05:07.6808269Z Generating XML reports...
2022-11-23T02:05:07.6808817Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123015252.xml
2022-11-23T02:05:07.6809507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6809962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6810536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6810992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6811465Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvyi8138p
2022-11-23T02:05:07.6812014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvyi8138p/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6812317Z 
2022-11-23T02:05:07.6812407Z Running tests...
2022-11-23T02:05:07.6812876Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6813420Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6813894Z   test_scatter_work_wait_cpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6814337Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6779
2022-11-23T02:05:07.6814787Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6780
2022-11-23T02:05:07.6815393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6815831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6816407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6817170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6817768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6818321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6818900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6819356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6819823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa6o9ivna
2022-11-23T02:05:07.6820353Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa6o9ivna/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6820871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6821373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9dftfw0o
2022-11-23T02:05:07.6821903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9dftfw0o/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6822419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6822909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:05:07.6823403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:05:07.6824047Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6824730Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6825648Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6826354Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6827187Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6827883Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6828203Z ok (4.155s)
2022-11-23T02:05:07.6828348Z 
2022-11-23T02:05:07.6828614Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6828921Z Ran 1 test in 4.155s
2022-11-23T02:05:07.6829082Z 
2022-11-23T02:05:07.6829173Z OK
2022-11-23T02:05:07.6829306Z 
2022-11-23T02:05:07.6829430Z Generating XML reports...
2022-11-23T02:05:07.6830070Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123015259.xml
2022-11-23T02:05:07.6830767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6831224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6831798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6832253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6832725Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw32nu234
2022-11-23T02:05:07.6833268Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw32nu234/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6833571Z 
2022-11-23T02:05:07.6833678Z Running tests...
2022-11-23T02:05:07.6834068Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6834603Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6835094Z   test_scatter_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6835616Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6984
2022-11-23T02:05:07.6836067Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6985
2022-11-23T02:05:07.6836694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6837156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6837716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6838191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6838770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6839201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6839780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6840359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6840828Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw59w7bs1
2022-11-23T02:05:07.6841349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw59w7bs1/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6841867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6842370Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps47ah0cn
2022-11-23T02:05:07.6842888Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps47ah0cn/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6843401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6843894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:05:07.6844396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:05:07.6845050Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6845739Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.6846664Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6847389Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6848292Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:05:07.6849019Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:05:07.6849350Z ok (5.737s)
2022-11-23T02:05:07.6849497Z 
2022-11-23T02:05:07.6849767Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6850076Z Ran 1 test in 5.738s
2022-11-23T02:05:07.6850238Z 
2022-11-23T02:05:07.6850330Z OK
2022-11-23T02:05:07.6850463Z 
2022-11-23T02:05:07.6850585Z Generating XML reports...
2022-11-23T02:05:07.6851126Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123015305.xml
2022-11-23T02:05:07.6851799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6852264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6852935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6853386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6853860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb65mbghz
2022-11-23T02:05:07.6854134Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb65mbghz/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6854155Z 
2022-11-23T02:05:07.6854262Z Running tests...
2022-11-23T02:05:07.6854531Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6854843Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6855074Z   test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6855436Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6855659Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7191
2022-11-23T02:05:07.6855876Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7192
2022-11-23T02:05:07.6856248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6856424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6857044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6857250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6857625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6857788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6858165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6858357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6858623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcyca7mth
2022-11-23T02:05:07.6858900Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcyca7mth/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6859131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6859394Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps1rct2hb
2022-11-23T02:05:07.6859666Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps1rct2hb/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6860110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6860207Z ok (6.248s)
2022-11-23T02:05:07.6860228Z 
2022-11-23T02:05:07.6860505Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6860617Z Ran 1 test in 6.248s
2022-11-23T02:05:07.6860637Z 
2022-11-23T02:05:07.6860729Z OK
2022-11-23T02:05:07.6860748Z 
2022-11-23T02:05:07.6860874Z Generating XML reports...
2022-11-23T02:05:07.6861345Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015313.xml
2022-11-23T02:05:07.6861722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6861897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6862262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6862456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6862711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9xz907hu
2022-11-23T02:05:07.6863053Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9xz907hu/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6863074Z 
2022-11-23T02:05:07.6863182Z Running tests...
2022-11-23T02:05:07.6863448Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6863759Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6863999Z   test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6864268Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6864470Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7406
2022-11-23T02:05:07.6864684Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7407
2022-11-23T02:05:07.6865061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6865238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6865620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6865812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6866177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6866355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6866712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6866899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6867164Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxhein3mb
2022-11-23T02:05:07.6867441Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxhein3mb/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6867697Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_oy0w970
2022-11-23T02:05:07.6867963Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_oy0w970/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6868192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6868421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6868503Z ok (6.217s)
2022-11-23T02:05:07.6868541Z 
2022-11-23T02:05:07.6868791Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6868903Z Ran 1 test in 6.217s
2022-11-23T02:05:07.6868923Z 
2022-11-23T02:05:07.6869014Z OK
2022-11-23T02:05:07.6869033Z 
2022-11-23T02:05:07.6869229Z Generating XML reports...
2022-11-23T02:05:07.6869709Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015322.xml
2022-11-23T02:05:07.6870083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6870260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6870645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6870944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6871202Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpip89kquz
2022-11-23T02:05:07.6871471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpip89kquz/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6871491Z 
2022-11-23T02:05:07.6871601Z Running tests...
2022-11-23T02:05:07.6871870Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6872246Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6872488Z   test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6872743Z DDP works as expected when layer is checkpointed only once. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6872945Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7621
2022-11-23T02:05:07.6873161Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7622
2022-11-23T02:05:07.6873534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6873712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6874094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6874286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6874657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6874831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6875202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6875375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6875632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwmd70rt1
2022-11-23T02:05:07.6875906Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwmd70rt1/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6876136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6876393Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprl6r_g3p
2022-11-23T02:05:07.6876662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprl6r_g3p/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6876886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6877125Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6877361Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6877575Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6877809Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6878778Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:05:07.6878902Z   warnings.warn(
2022-11-23T02:05:07.6879816Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:05:07.6879927Z   warnings.warn(
2022-11-23T02:05:07.6880160Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6880389Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6880623Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6880856Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6881131Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6881359Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6881583Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6881814Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6881916Z ok (6.331s)
2022-11-23T02:05:07.6881936Z 
2022-11-23T02:05:07.6882208Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6882320Z Ran 1 test in 6.331s
2022-11-23T02:05:07.6882339Z 
2022-11-23T02:05:07.6882430Z OK
2022-11-23T02:05:07.6882448Z 
2022-11-23T02:05:07.6882554Z Generating XML reports...
2022-11-23T02:05:07.6883027Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015331.xml
2022-11-23T02:05:07.6883404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6883589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6883970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6884163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6884422Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp52h_omfh
2022-11-23T02:05:07.6884697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp52h_omfh/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6884716Z 
2022-11-23T02:05:07.6884824Z Running tests...
2022-11-23T02:05:07.6885074Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6885393Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6885638Z   test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6885889Z DDP works as expected when layer is checkpointed only once. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6886109Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7836
2022-11-23T02:05:07.6886327Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7837
2022-11-23T02:05:07.6886698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6886875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6887235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6887474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6887848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6888025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6888398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6888589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6888848Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf99iiuup
2022-11-23T02:05:07.6889119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf99iiuup/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6889349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6889594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzo99tfnf
2022-11-23T02:05:07.6889867Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzo99tfnf/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6890151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6890389Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6890624Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6890977Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6891208Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6892128Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:05:07.6892244Z   warnings.warn(
2022-11-23T02:05:07.6893152Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:05:07.6893250Z   warnings.warn(
2022-11-23T02:05:07.6893486Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6893721Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6893954Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6894189Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6894419Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6894644Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6894873Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6895085Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6895187Z ok (6.346s)
2022-11-23T02:05:07.6895207Z 
2022-11-23T02:05:07.6895481Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6895595Z Ran 1 test in 6.346s
2022-11-23T02:05:07.6895615Z 
2022-11-23T02:05:07.6895702Z OK
2022-11-23T02:05:07.6895721Z 
2022-11-23T02:05:07.6895842Z Generating XML reports...
2022-11-23T02:05:07.6896361Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015340.xml
2022-11-23T02:05:07.6896978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6897173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6897546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6897743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6898005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_4p3zl6h
2022-11-23T02:05:07.6898274Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_4p3zl6h/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6898295Z 
2022-11-23T02:05:07.6898402Z Running tests...
2022-11-23T02:05:07.6898669Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6898984Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6899251Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6899679Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6899898Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8051
2022-11-23T02:05:07.6900114Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8052
2022-11-23T02:05:07.6900489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6900664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6901070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6901315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6901689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6901869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6902234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6902424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6902684Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpumru3f28
2022-11-23T02:05:07.6902958Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpumru3f28/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6903189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6903446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp6o5_hqm
2022-11-23T02:05:07.6903726Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp6o5_hqm/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6903962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6904182Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6904420Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6904653Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6904885Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6904994Z ok (6.230s)
2022-11-23T02:05:07.6905014Z 
2022-11-23T02:05:07.6905290Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6905406Z Ran 1 test in 6.231s
2022-11-23T02:05:07.6905425Z 
2022-11-23T02:05:07.6905523Z OK
2022-11-23T02:05:07.6905542Z 
2022-11-23T02:05:07.6905747Z Generating XML reports...
2022-11-23T02:05:07.6906217Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015348.xml
2022-11-23T02:05:07.6906598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6906776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6907160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6907357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6907616Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_xirvliq
2022-11-23T02:05:07.6907893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_xirvliq/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6907914Z 
2022-11-23T02:05:07.6908026Z Running tests...
2022-11-23T02:05:07.6908278Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6908655Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6908928Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6909285Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6909508Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8266
2022-11-23T02:05:07.6909727Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8267
2022-11-23T02:05:07.6910102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6910284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6910677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6910851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6911225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6911403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6911775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6911970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6912235Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx234xb11
2022-11-23T02:05:07.6912510Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx234xb11/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6912744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6912984Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_szx8kwt
2022-11-23T02:05:07.6913262Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_szx8kwt/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6913492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6913734Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6913977Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6914212Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6914445Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6914551Z ok (6.365s)
2022-11-23T02:05:07.6914570Z 
2022-11-23T02:05:07.6914842Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6914994Z Ran 1 test in 6.365s
2022-11-23T02:05:07.6915016Z 
2022-11-23T02:05:07.6915117Z OK
2022-11-23T02:05:07.6915140Z 
2022-11-23T02:05:07.6915273Z Generating XML reports...
2022-11-23T02:05:07.6915752Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015357.xml
2022-11-23T02:05:07.6916128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6916311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6916697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6916890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6917128Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkmv1jnkv
2022-11-23T02:05:07.6917405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkmv1jnkv/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6917425Z 
2022-11-23T02:05:07.6917538Z Running tests...
2022-11-23T02:05:07.6917862Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6918185Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6918432Z   test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6918817Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6919040Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8481
2022-11-23T02:05:07.6919267Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8482
2022-11-23T02:05:07.6919619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6919802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6920186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6920386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6920754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6920933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6921314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6921507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6921748Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu58tybph
2022-11-23T02:05:07.6922025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu58tybph/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6922293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0mvx_aoh
2022-11-23T02:05:07.6922571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0mvx_aoh/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6922808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6923040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6923283Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6923520Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6924361Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:05:07.6925161Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:05:07.6925402Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6925648Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6925731Z ok (6.435s)
2022-11-23T02:05:07.6925751Z 
2022-11-23T02:05:07.6926091Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6926208Z Ran 1 test in 6.435s
2022-11-23T02:05:07.6926227Z 
2022-11-23T02:05:07.6926324Z OK
2022-11-23T02:05:07.6926343Z 
2022-11-23T02:05:07.6926474Z Generating XML reports...
2022-11-23T02:05:07.6926952Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015406.xml
2022-11-23T02:05:07.6927327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6927509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6927895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6928073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6928340Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7apzq4li
2022-11-23T02:05:07.6928621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7apzq4li/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6928641Z 
2022-11-23T02:05:07.6928754Z Running tests...
2022-11-23T02:05:07.6929027Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6929346Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6929593Z   test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6929974Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6930175Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8696
2022-11-23T02:05:07.6930403Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8697
2022-11-23T02:05:07.6930780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6930967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6931354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6931551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6931924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6932101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6932478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6932653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6932980Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5vuijqfz
2022-11-23T02:05:07.6933268Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5vuijqfz/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6933529Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzb1vvy5x
2022-11-23T02:05:07.6933806Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzb1vvy5x/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6934038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6934264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6934371Z ok (6.234s)
2022-11-23T02:05:07.6934391Z 
2022-11-23T02:05:07.6934647Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6934772Z Ran 1 test in 6.234s
2022-11-23T02:05:07.6934791Z 
2022-11-23T02:05:07.6934887Z OK
2022-11-23T02:05:07.6934906Z 
2022-11-23T02:05:07.6935042Z Generating XML reports...
2022-11-23T02:05:07.6935518Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015415.xml
2022-11-23T02:05:07.6935954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6936133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6936519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6936932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6937193Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ppf94h8
2022-11-23T02:05:07.6937472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ppf94h8/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6937493Z 
2022-11-23T02:05:07.6937607Z Running tests...
2022-11-23T02:05:07.6937892Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6938214Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6938458Z   test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6938735Z Checkpointing should work with static graph in the case of checkpointing ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6938956Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8911
2022-11-23T02:05:07.6939154Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8912
2022-11-23T02:05:07.6939536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6939717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6940194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6940397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6940774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6940955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6941331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6941524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6941767Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjt6lxjo9
2022-11-23T02:05:07.6942045Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjt6lxjo9/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6942278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6942625Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfprd_zgw
2022-11-23T02:05:07.6942919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfprd_zgw/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6943157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6943397Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6943636Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6943853Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6944092Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6944197Z ok (6.331s)
2022-11-23T02:05:07.6944218Z 
2022-11-23T02:05:07.6944500Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6944621Z Ran 1 test in 6.331s
2022-11-23T02:05:07.6944641Z 
2022-11-23T02:05:07.6944738Z OK
2022-11-23T02:05:07.6944822Z 
2022-11-23T02:05:07.6944956Z Generating XML reports...
2022-11-23T02:05:07.6945431Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015424.xml
2022-11-23T02:05:07.6945812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6945971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6946361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6946556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6946818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc45qoefh
2022-11-23T02:05:07.6947100Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc45qoefh/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6947120Z 
2022-11-23T02:05:07.6947234Z Running tests...
2022-11-23T02:05:07.6947501Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6947815Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6948057Z   test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6948334Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6948559Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9126
2022-11-23T02:05:07.6948780Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9127
2022-11-23T02:05:07.6949156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6949340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6949727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6949967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6950339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6950495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6950879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6951074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6951339Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqpul39fc
2022-11-23T02:05:07.6951668Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqpul39fc/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6951937Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp363ti81c
2022-11-23T02:05:07.6952208Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp363ti81c/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6952441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6952651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6953442Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:05:07.6954256Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:05:07.6955235Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:05:07.6955358Z   warnings.warn(
2022-11-23T02:05:07.6956251Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:05:07.6956372Z   warnings.warn(
2022-11-23T02:05:07.6956615Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6956858Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6957100Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6957340Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6957444Z ok (6.345s)
2022-11-23T02:05:07.6957466Z 
2022-11-23T02:05:07.6957741Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6957862Z Ran 1 test in 6.345s
2022-11-23T02:05:07.6957881Z 
2022-11-23T02:05:07.6957955Z OK
2022-11-23T02:05:07.6957974Z 
2022-11-23T02:05:07.6958105Z Generating XML reports...
2022-11-23T02:05:07.6958579Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015432.xml
2022-11-23T02:05:07.6958955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6959138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6959527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6959725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6960093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpruj9kcaj
2022-11-23T02:05:07.6960381Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpruj9kcaj/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6960404Z 
2022-11-23T02:05:07.6960495Z Running tests...
2022-11-23T02:05:07.6960769Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6961088Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6961351Z   test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6961630Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6961848Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9341
2022-11-23T02:05:07.6962065Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9342
2022-11-23T02:05:07.6962442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6962653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6963042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6963238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6963606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6963784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6964158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6964349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6964615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4k7kfq33
2022-11-23T02:05:07.6964891Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4k7kfq33/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6965134Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc6cvqq0q
2022-11-23T02:05:07.6965414Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc6cvqq0q/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6965649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6965883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6966810Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:05:07.6966932Z   warnings.warn(
2022-11-23T02:05:07.6967846Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:05:07.6967961Z   warnings.warn(
2022-11-23T02:05:07.6968206Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6968442Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6968659Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6968903Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6969057Z ok (6.397s)
2022-11-23T02:05:07.6969082Z 
2022-11-23T02:05:07.6969361Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6969480Z Ran 1 test in 6.397s
2022-11-23T02:05:07.6969500Z 
2022-11-23T02:05:07.6969595Z OK
2022-11-23T02:05:07.6969613Z 
2022-11-23T02:05:07.6969742Z Generating XML reports...
2022-11-23T02:05:07.6970213Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015441.xml
2022-11-23T02:05:07.6970568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6970748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6971131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6971327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6971591Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6flnpqmh
2022-11-23T02:05:07.6971924Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6flnpqmh/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6971944Z 
2022-11-23T02:05:07.6972056Z Running tests...
2022-11-23T02:05:07.6972324Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6972640Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6972886Z   test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6973129Z Test that checkpointing with weight sharing works. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6973351Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9556
2022-11-23T02:05:07.6973571Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9557
2022-11-23T02:05:07.6973956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6974137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6974521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6974723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6975074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6975253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6975637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6975833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6976105Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp37jzz3x
2022-11-23T02:05:07.6976386Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp37jzz3x/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6976648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp77go4_en
2022-11-23T02:05:07.6977202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp77go4_en/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6977449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6977662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6977905Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6978146Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6978383Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6978696Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6978817Z ok (6.335s)
2022-11-23T02:05:07.6978837Z 
2022-11-23T02:05:07.6979119Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6979236Z Ran 1 test in 6.335s
2022-11-23T02:05:07.6979256Z 
2022-11-23T02:05:07.6979329Z OK
2022-11-23T02:05:07.6979348Z 
2022-11-23T02:05:07.6979479Z Generating XML reports...
2022-11-23T02:05:07.6979958Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015450.xml
2022-11-23T02:05:07.6980331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6980513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6980902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6981102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6981432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8wrd94ni
2022-11-23T02:05:07.6981712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8wrd94ni/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6981732Z 
2022-11-23T02:05:07.6981823Z Running tests...
2022-11-23T02:05:07.6982104Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6982422Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6982690Z   test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6982933Z Test that checkpointing with weight sharing works. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6983157Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9771
2022-11-23T02:05:07.6983385Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9772
2022-11-23T02:05:07.6983769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6983926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6984307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6984505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6984879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6985059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6985437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6985635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6985899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphi9eb1y4
2022-11-23T02:05:07.6986181Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphi9eb1y4/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6986420Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi89pffun
2022-11-23T02:05:07.6986702Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi89pffun/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6986937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6987173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6987412Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6987654Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6987945Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6988200Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6988414Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6988643Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6988879Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6989119Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.6989224Z ok (6.347s)
2022-11-23T02:05:07.6989244Z 
2022-11-23T02:05:07.6989517Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6989636Z Ran 1 test in 6.347s
2022-11-23T02:05:07.6989656Z 
2022-11-23T02:05:07.6989751Z OK
2022-11-23T02:05:07.6989769Z 
2022-11-23T02:05:07.6989878Z Generating XML reports...
2022-11-23T02:05:07.6990352Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015459.xml
2022-11-23T02:05:07.6990792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6990972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6991355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6991553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6991817Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ps0g9_g
2022-11-23T02:05:07.6992092Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ps0g9_g/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6992111Z 
2022-11-23T02:05:07.6992223Z Running tests...
2022-11-23T02:05:07.6992480Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6992800Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.6993030Z   test_ddp_comm_hook_future_passing_cpu (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.6993304Z This unit test verifies whether the Future object is passed properly. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.6993526Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9986
2022-11-23T02:05:07.6993746Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9987
2022-11-23T02:05:07.6994126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6994305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6994672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6994870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6995247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6995423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6995803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.6995996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.6996263Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph_s01h40
2022-11-23T02:05:07.6996537Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph_s01h40/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6996797Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpafdrgdow
2022-11-23T02:05:07.6997116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpafdrgdow/_remote_module_non_scriptable.py
2022-11-23T02:05:07.6997368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.6997606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.6997711Z ok (4.129s)
2022-11-23T02:05:07.6997731Z 
2022-11-23T02:05:07.6998003Z ----------------------------------------------------------------------
2022-11-23T02:05:07.6998119Z Ran 1 test in 4.129s
2022-11-23T02:05:07.6998139Z 
2022-11-23T02:05:07.6998237Z OK
2022-11-23T02:05:07.6998256Z 
2022-11-23T02:05:07.6998385Z Generating XML reports...
2022-11-23T02:05:07.6998832Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015508.xml
2022-11-23T02:05:07.6999207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.6999390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.6999840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7000036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7000296Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmqz_7oxh
2022-11-23T02:05:07.7000572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmqz_7oxh/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7000591Z 
2022-11-23T02:05:07.7000704Z Running tests...
2022-11-23T02:05:07.7000975Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7001270Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7001510Z   test_ddp_comm_hook_future_passing_gpu_gloo (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.7001813Z This unit test verifies whether the Future object is passed properly using gloo backend. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7002042Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10199
2022-11-23T02:05:07.7002269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10200
2022-11-23T02:05:07.7002651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7002830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7003212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7003411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7003761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7003940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7004316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7004515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7004778Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps57y0s2n
2022-11-23T02:05:07.7005051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps57y0s2n/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7005288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7005551Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm15t_8h2
2022-11-23T02:05:07.7005799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm15t_8h2/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7006083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7006196Z ok (5.746s)
2022-11-23T02:05:07.7006216Z 
2022-11-23T02:05:07.7006494Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7006610Z Ran 1 test in 5.746s
2022-11-23T02:05:07.7006629Z 
2022-11-23T02:05:07.7006724Z OK
2022-11-23T02:05:07.7006743Z 
2022-11-23T02:05:07.7006870Z Generating XML reports...
2022-11-23T02:05:07.7007341Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015514.xml
2022-11-23T02:05:07.7007718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7007875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7008258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7008456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7008719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbcjgaxtw
2022-11-23T02:05:07.7009054Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbcjgaxtw/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7009073Z 
2022-11-23T02:05:07.7009186Z Running tests...
2022-11-23T02:05:07.7009457Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7009777Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7009983Z   test_ddp_comm_hook_register_just_once (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.7010273Z DDP communication hook can only be registered once. This test validates whether ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7010496Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10414
2022-11-23T02:05:07.7010720Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10415
2022-11-23T02:05:07.7011104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7011287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7011667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7011864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7012237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7012394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7012779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7012973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7013236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpavfyxe7n
2022-11-23T02:05:07.7013519Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpavfyxe7n/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7013756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7014018Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnch0dasu
2022-11-23T02:05:07.7014297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnch0dasu/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7014507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7014615Z ok (4.149s)
2022-11-23T02:05:07.7014634Z 
2022-11-23T02:05:07.7014905Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7015022Z Ran 1 test in 4.149s
2022-11-23T02:05:07.7015042Z 
2022-11-23T02:05:07.7015140Z OK
2022-11-23T02:05:07.7015159Z 
2022-11-23T02:05:07.7015338Z Generating XML reports...
2022-11-23T02:05:07.7015822Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015522.xml
2022-11-23T02:05:07.7016204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7016365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7017001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7017213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7017475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_0xyy7ot
2022-11-23T02:05:07.7017747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_0xyy7ot/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7017767Z 
2022-11-23T02:05:07.7017880Z Running tests...
2022-11-23T02:05:07.7018159Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7018573Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7018805Z   test_ddp_comm_hook_sparse_gradients (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.7019066Z Runs "test_sparse_gradients" unit test with DDP communication hook. We define a ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7019292Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10619
2022-11-23T02:05:07.7019515Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10620
2022-11-23T02:05:07.7019891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7020069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7020459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7020655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7021028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7021184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7021564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7021759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7022021Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7gxvya82
2022-11-23T02:05:07.7022294Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7gxvya82/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7022529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7022791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ss5dcm6
2022-11-23T02:05:07.7023073Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ss5dcm6/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7023310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7023393Z ok (4.056s)
2022-11-23T02:05:07.7023413Z 
2022-11-23T02:05:07.7023688Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7023803Z Ran 1 test in 4.056s
2022-11-23T02:05:07.7023823Z 
2022-11-23T02:05:07.7023920Z OK
2022-11-23T02:05:07.7023939Z 
2022-11-23T02:05:07.7024067Z Generating XML reports...
2022-11-23T02:05:07.7024538Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015529.xml
2022-11-23T02:05:07.7024976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7025166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7025537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7025733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7025997Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdsklx8a4
2022-11-23T02:05:07.7026275Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdsklx8a4/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7026295Z 
2022-11-23T02:05:07.7026406Z Running tests...
2022-11-23T02:05:07.7026675Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7026993Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7027213Z   test_ddp_invalid_comm_hook_init (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.7027496Z This unit test makes sure that register_comm_hook properly checks the format ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7027757Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10894
2022-11-23T02:05:07.7027978Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10895
2022-11-23T02:05:07.7028357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7028535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7028917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7029113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7029483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7029665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7030019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7030213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7030482Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwcpx6fwm
2022-11-23T02:05:07.7030743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwfdvpo2k
2022-11-23T02:05:07.7031021Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwcpx6fwm/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7031293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwfdvpo2k/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7031529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7031765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7031874Z ok (4.157s)
2022-11-23T02:05:07.7031894Z 
2022-11-23T02:05:07.7032147Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7032264Z Ran 1 test in 4.157s
2022-11-23T02:05:07.7032284Z 
2022-11-23T02:05:07.7032382Z OK
2022-11-23T02:05:07.7032401Z 
2022-11-23T02:05:07.7032529Z Generating XML reports...
2022-11-23T02:05:07.7032999Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015536.xml
2022-11-23T02:05:07.7033369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7033549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7033934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7034109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7034440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwyt1jw7w
2022-11-23T02:05:07.7034729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwyt1jw7w/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7034749Z 
2022-11-23T02:05:07.7034861Z Running tests...
2022-11-23T02:05:07.7035132Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7035451Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7035683Z   test_ddp_invalid_comm_hook_return_type (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.7035973Z This test checks whether return annotation checked properly if defined. It also ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7036196Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11099
2022-11-23T02:05:07.7036397Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11100
2022-11-23T02:05:07.7036781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7037015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7037407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7037601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7037969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7038147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7038522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7038692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7038958Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuqex9t7m
2022-11-23T02:05:07.7039235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuqex9t7m/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7039502Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkn57dbo7
2022-11-23T02:05:07.7039777Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkn57dbo7/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7040103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7040348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7040457Z ok (4.173s)
2022-11-23T02:05:07.7040478Z 
2022-11-23T02:05:07.7040753Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7040845Z Ran 1 test in 4.173s
2022-11-23T02:05:07.7040864Z 
2022-11-23T02:05:07.7040959Z OK
2022-11-23T02:05:07.7040978Z 
2022-11-23T02:05:07.7041111Z Generating XML reports...
2022-11-23T02:05:07.7041583Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015542.xml
2022-11-23T02:05:07.7041964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7042143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7042529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7042727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7042965Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp659fupur
2022-11-23T02:05:07.7043241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp659fupur/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7043261Z 
2022-11-23T02:05:07.7043371Z Running tests...
2022-11-23T02:05:07.7043738Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7044075Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7044340Z   test_find_unused_parameters_when_unused_parameters_empty (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.7044617Z An empty unused_parameters array does not imply find_unused_parameters = ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7044844Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11312
2022-11-23T02:05:07.7045068Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11313
2022-11-23T02:05:07.7045425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7045603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7045993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7046187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7046613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7046793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7047171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7047362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7047604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6s7jszmj
2022-11-23T02:05:07.7047884Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6s7jszmj/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7048144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3hpzzk86
2022-11-23T02:05:07.7048423Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3hpzzk86/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7048660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7048891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7049674Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:05:07.7049783Z ok (5.717s)
2022-11-23T02:05:07.7049803Z 
2022-11-23T02:05:07.7050085Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7050200Z Ran 1 test in 5.717s
2022-11-23T02:05:07.7050222Z 
2022-11-23T02:05:07.7050316Z OK
2022-11-23T02:05:07.7050336Z 
2022-11-23T02:05:07.7050443Z Generating XML reports...
2022-11-23T02:05:07.7050917Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015549.xml
2022-11-23T02:05:07.7051293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7051474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7051861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7052059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7052373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppafpgp47
2022-11-23T02:05:07.7052657Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppafpgp47/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7052681Z 
2022-11-23T02:05:07.7052770Z Running tests...
2022-11-23T02:05:07.7053044Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7053355Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7053644Z   test_global_local_unused_params_grad (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7053864Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11527
2022-11-23T02:05:07.7054088Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11528
2022-11-23T02:05:07.7054457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7054639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7055017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7055246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7055615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7055792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7056164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7056354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7056614Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzhvqsuk6
2022-11-23T02:05:07.7057187Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzhvqsuk6/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7057457Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbzl3cubr
2022-11-23T02:05:07.7057713Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbzl3cubr/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7057945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7058176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7058277Z ok (5.735s)
2022-11-23T02:05:07.7058297Z 
2022-11-23T02:05:07.7058572Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7058682Z Ran 1 test in 5.735s
2022-11-23T02:05:07.7058702Z 
2022-11-23T02:05:07.7058794Z OK
2022-11-23T02:05:07.7058813Z 
2022-11-23T02:05:07.7058936Z Generating XML reports...
2022-11-23T02:05:07.7059409Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015557.xml
2022-11-23T02:05:07.7059764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7059943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7060326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7060516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7060769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmj3cjb69
2022-11-23T02:05:07.7061036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmj3cjb69/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7061056Z 
2022-11-23T02:05:07.7061166Z Running tests...
2022-11-23T02:05:07.7061433Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7061727Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7062135Z   test_global_local_unused_params_grad_with_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7062374Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11742
2022-11-23T02:05:07.7062593Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11743
2022-11-23T02:05:07.7062971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7063146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7063523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7063715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7064082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7064238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7064688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7064876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7065135Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdgsyl87s
2022-11-23T02:05:07.7065409Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdgsyl87s/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7065664Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvfze_1rt
2022-11-23T02:05:07.7065931Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvfze_1rt/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7066163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7066378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7066479Z ok (5.749s)
2022-11-23T02:05:07.7066501Z 
2022-11-23T02:05:07.7066772Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7066882Z Ran 1 test in 5.750s
2022-11-23T02:05:07.7066902Z 
2022-11-23T02:05:07.7066994Z OK
2022-11-23T02:05:07.7067014Z 
2022-11-23T02:05:07.7067137Z Generating XML reports...
2022-11-23T02:05:07.7067603Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015605.xml
2022-11-23T02:05:07.7067975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7068153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7068515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7068707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7068961Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_kirn7c4
2022-11-23T02:05:07.7069233Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_kirn7c4/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7069253Z 
2022-11-23T02:05:07.7069364Z Running tests...
2022-11-23T02:05:07.7069629Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7069941Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7070256Z   test_global_local_unused_params_grad_with_static_graph (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7070460Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11957
2022-11-23T02:05:07.7070680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11958
2022-11-23T02:05:07.7071106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7071291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7071673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7071864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7072307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7072485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7072863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7073033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7073299Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpip0isnlq
2022-11-23T02:05:07.7073574Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpip0isnlq/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7073863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7074121Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprn88g_ex
2022-11-23T02:05:07.7074391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprn88g_ex/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7074619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7075540Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:05:07.7075653Z   warnings.warn(
2022-11-23T02:05:07.7076564Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:05:07.7076673Z   warnings.warn(
2022-11-23T02:05:07.7076755Z ok (5.759s)
2022-11-23T02:05:07.7076775Z 
2022-11-23T02:05:07.7077042Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7077152Z Ran 1 test in 5.760s
2022-11-23T02:05:07.7077172Z 
2022-11-23T02:05:07.7077262Z OK
2022-11-23T02:05:07.7077281Z 
2022-11-23T02:05:07.7077405Z Generating XML reports...
2022-11-23T02:05:07.7077875Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015613.xml
2022-11-23T02:05:07.7078247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7078425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7078788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7078978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7079236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1fkwdxcv
2022-11-23T02:05:07.7079509Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1fkwdxcv/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7079529Z 
2022-11-23T02:05:07.7079635Z Running tests...
2022-11-23T02:05:07.7079899Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7080267Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7080586Z   test_gloo_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7080793Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12172
2022-11-23T02:05:07.7081014Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12173
2022-11-23T02:05:07.7081391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7081567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7081948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7082140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7082503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7082675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7083111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7083282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7083542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppf5982i4
2022-11-23T02:05:07.7083817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppf5982i4/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7084053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7084311Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_qupw8s7
2022-11-23T02:05:07.7084586Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_qupw8s7/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7084821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7085065Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.7085284Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.7085391Z ok (6.165s)
2022-11-23T02:05:07.7085411Z 
2022-11-23T02:05:07.7085683Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7085799Z Ran 1 test in 6.165s
2022-11-23T02:05:07.7085818Z 
2022-11-23T02:05:07.7085914Z OK
2022-11-23T02:05:07.7085933Z 
2022-11-23T02:05:07.7086061Z Generating XML reports...
2022-11-23T02:05:07.7086528Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015622.xml
2022-11-23T02:05:07.7086900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7087084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7087453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7087645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7087902Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv8z5fc5a
2022-11-23T02:05:07.7088174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv8z5fc5a/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7088193Z 
2022-11-23T02:05:07.7088303Z Running tests...
2022-11-23T02:05:07.7088566Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7088876Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7089246Z   test_gloo_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7089458Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12387
2022-11-23T02:05:07.7089680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12388
2022-11-23T02:05:07.7090058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7090237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7090616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7090807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7091171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7091343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7091722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7091946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7092206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3oc2g0pq
2022-11-23T02:05:07.7092479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3oc2g0pq/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7092735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp35td6_t8
2022-11-23T02:05:07.7093003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp35td6_t8/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7093237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7093464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7093702Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.7093940Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.7094026Z ok (6.195s)
2022-11-23T02:05:07.7094046Z 
2022-11-23T02:05:07.7094319Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7094434Z Ran 1 test in 6.195s
2022-11-23T02:05:07.7094453Z 
2022-11-23T02:05:07.7094548Z OK
2022-11-23T02:05:07.7094567Z 
2022-11-23T02:05:07.7094690Z Generating XML reports...
2022-11-23T02:05:07.7095156Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015630.xml
2022-11-23T02:05:07.7095528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7095703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7096066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7096257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7096520Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqxqkfbep
2022-11-23T02:05:07.7097053Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqxqkfbep/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7097074Z 
2022-11-23T02:05:07.7097190Z Running tests...
2022-11-23T02:05:07.7097466Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7097780Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7098057Z   test_gloo_backend_2gpu_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7098260Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12602
2022-11-23T02:05:07.7098555Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12603
2022-11-23T02:05:07.7098942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7099120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7099500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7099692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7100051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7100227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7100603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7100773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7101034Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkgz35kj0
2022-11-23T02:05:07.7101429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkgz35kj0/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7101658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7101914Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptrh0rdty
2022-11-23T02:05:07.7102182Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptrh0rdty/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7102407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7102767Z [W logger.cpp:318] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-11-23T02:05:07.7103119Z [W logger.cpp:318] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-11-23T02:05:07.7103342Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.7103583Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.7103685Z ok (7.951s)
2022-11-23T02:05:07.7103704Z 
2022-11-23T02:05:07.7103967Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7104082Z Ran 1 test in 7.951s
2022-11-23T02:05:07.7104102Z 
2022-11-23T02:05:07.7104198Z OK
2022-11-23T02:05:07.7104217Z 
2022-11-23T02:05:07.7104345Z Generating XML reports...
2022-11-23T02:05:07.7104813Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015639.xml
2022-11-23T02:05:07.7105168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7105347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7105738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7105937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7106198Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6hjyexev
2022-11-23T02:05:07.7106480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6hjyexev/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7106500Z 
2022-11-23T02:05:07.7106612Z Running tests...
2022-11-23T02:05:07.7106882Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7107199Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7107475Z   test_gloo_backend_4gpu_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7107700Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12819
2022-11-23T02:05:07.7107972Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12820
2022-11-23T02:05:07.7108364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7108543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7108924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7109122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7109489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7109698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7110081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7110280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7110548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgsl51sw_
2022-11-23T02:05:07.7110878Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgsl51sw_/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7111111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7111368Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv_t1scv7
2022-11-23T02:05:07.7111643Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv_t1scv7/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7111879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7112011Z skip: Need at least 8 CUDA devices (4.127s)
2022-11-23T02:05:07.7112031Z 
2022-11-23T02:05:07.7112308Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7112423Z Ran 1 test in 4.127s
2022-11-23T02:05:07.7112446Z 
2022-11-23T02:05:07.7112555Z OK (skipped=1)
2022-11-23T02:05:07.7112577Z 
2022-11-23T02:05:07.7112705Z Generating XML reports...
2022-11-23T02:05:07.7113176Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015649.xml
2022-11-23T02:05:07.7113554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7113733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7114096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7114291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7114553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl3zyn5nb
2022-11-23T02:05:07.7114831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl3zyn5nb/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7114852Z 
2022-11-23T02:05:07.7114964Z Running tests...
2022-11-23T02:05:07.7115241Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7115557Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7115841Z   test_gloo_backend_cpu_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7116066Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13018
2022-11-23T02:05:07.7116265Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13019
2022-11-23T02:05:07.7116643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7116824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7117255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7117457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7117834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7118013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7118395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7118567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7118835Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnmrn9dt_
2022-11-23T02:05:07.7119111Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnmrn9dt_/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7119375Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkeh3xd7w
2022-11-23T02:05:07.7119649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkeh3xd7w/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7119936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7120168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7120411Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.7120647Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.7120730Z ok (4.125s)
2022-11-23T02:05:07.7120749Z 
2022-11-23T02:05:07.7121023Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7121140Z Ran 1 test in 4.125s
2022-11-23T02:05:07.7121160Z 
2022-11-23T02:05:07.7121255Z OK
2022-11-23T02:05:07.7121275Z 
2022-11-23T02:05:07.7121400Z Generating XML reports...
2022-11-23T02:05:07.7121878Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015656.xml
2022-11-23T02:05:07.7122256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7122435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7122797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7122992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7123254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkh9hypkx
2022-11-23T02:05:07.7123531Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkh9hypkx/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7123550Z 
2022-11-23T02:05:07.7123662Z Running tests...
2022-11-23T02:05:07.7123931Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7124253Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7124558Z   test_gloo_backend_cpu_module_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7124784Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13231
2022-11-23T02:05:07.7124985Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13232
2022-11-23T02:05:07.7125362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7125544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7125929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7126127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7126570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7126759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7127144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7127314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7127582Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg9a0aart
2022-11-23T02:05:07.7127866Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg9a0aart/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7128131Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzro0m6v2
2022-11-23T02:05:07.7128408Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzro0m6v2/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7128644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7128877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7129188Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.7129419Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.7129503Z ok (4.218s)
2022-11-23T02:05:07.7129523Z 
2022-11-23T02:05:07.7129796Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7129916Z Ran 1 test in 4.218s
2022-11-23T02:05:07.7129936Z 
2022-11-23T02:05:07.7130031Z OK
2022-11-23T02:05:07.7130049Z 
2022-11-23T02:05:07.7130178Z Generating XML reports...
2022-11-23T02:05:07.7130646Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015703.xml
2022-11-23T02:05:07.7131024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7131207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7131572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7131766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7132025Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr1wjj_1v
2022-11-23T02:05:07.7132302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr1wjj_1v/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7132321Z 
2022-11-23T02:05:07.7132433Z Running tests...
2022-11-23T02:05:07.7132703Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7133019Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7133223Z   test_ignored_output (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.7133486Z Test that the output of a model can be ignored and that there is no ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7133693Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13444
2022-11-23T02:05:07.7133915Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13445
2022-11-23T02:05:07.7134294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7134473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7134855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7135051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7135410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7135647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7136013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7136209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7136469Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8xks8fl6
2022-11-23T02:05:07.7136988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8xks8fl6/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7137239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7137501Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprz3ut2ak
2022-11-23T02:05:07.7137774Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprz3ut2ak/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7138007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7138248Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.7138557Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.7138664Z ok (4.141s)
2022-11-23T02:05:07.7138686Z 
2022-11-23T02:05:07.7138970Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7139086Z Ran 1 test in 4.141s
2022-11-23T02:05:07.7139106Z 
2022-11-23T02:05:07.7139202Z OK
2022-11-23T02:05:07.7139221Z 
2022-11-23T02:05:07.7139349Z Generating XML reports...
2022-11-23T02:05:07.7139823Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015709.xml
2022-11-23T02:05:07.7140289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7140451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7140845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7141043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7141307Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwoy8xgzn
2022-11-23T02:05:07.7141587Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwoy8xgzn/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7141607Z 
2022-11-23T02:05:07.7141723Z Running tests...
2022-11-23T02:05:07.7141996Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7142312Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7142554Z   test_ignored_output_with_unused_parameters (__main__.DistributedDataParallelTest)
2022-11-23T02:05:07.7142798Z Test that the output of a model can be ignored and that there is no ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7143021Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13719
2022-11-23T02:05:07.7143250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13720
2022-11-23T02:05:07.7143629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7143810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7144189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7144383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7144746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7144901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7145350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7145561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7145822Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzbbzb_gx
2022-11-23T02:05:07.7146098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzbbzb_gx/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7146334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7146596Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpspskqx5n
2022-11-23T02:05:07.7146878Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpspskqx5n/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7147112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7147194Z ok (4.161s)
2022-11-23T02:05:07.7147214Z 
2022-11-23T02:05:07.7147496Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7147614Z Ran 1 test in 4.161s
2022-11-23T02:05:07.7147682Z 
2022-11-23T02:05:07.7147780Z OK
2022-11-23T02:05:07.7147799Z 
2022-11-23T02:05:07.7147933Z Generating XML reports...
2022-11-23T02:05:07.7148402Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015716.xml
2022-11-23T02:05:07.7148780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7148961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7149324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7149518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7149783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi39f1fj7
2022-11-23T02:05:07.7150058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi39f1fj7/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7150082Z 
2022-11-23T02:05:07.7150194Z Running tests...
2022-11-23T02:05:07.7150466Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7150788Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7151075Z   test_ignored_sharded_tensor (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7151298Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13994
2022-11-23T02:05:07.7151500Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13995
2022-11-23T02:05:07.7151877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7152065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7152451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7152652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7153028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7153206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7153585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7153756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7154022Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo3sr3d5b
2022-11-23T02:05:07.7154297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo3sr3d5b/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7154613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2b8e7qzm
2022-11-23T02:05:07.7154897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2b8e7qzm/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7155132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7155369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7155620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:05:07.7156034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.7156261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:05:07.7156673Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.7156780Z ok (5.729s)
2022-11-23T02:05:07.7156799Z 
2022-11-23T02:05:07.7157122Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7157240Z Ran 1 test in 5.729s
2022-11-23T02:05:07.7157260Z 
2022-11-23T02:05:07.7157356Z OK
2022-11-23T02:05:07.7157374Z 
2022-11-23T02:05:07.7157504Z Generating XML reports...
2022-11-23T02:05:07.7157970Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015723.xml
2022-11-23T02:05:07.7158327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7158504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7158891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7159090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7159355Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn09vk8mg
2022-11-23T02:05:07.7159633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn09vk8mg/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7159653Z 
2022-11-23T02:05:07.7159767Z Running tests...
2022-11-23T02:05:07.7160042Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7160360Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7160620Z   test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7160843Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14201
2022-11-23T02:05:07.7161066Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14202
2022-11-23T02:05:07.7161448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7161630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7162017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7162217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7162588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7162742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7163128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7163321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7163584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpujw6b22p
2022-11-23T02:05:07.7163913Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpujw6b22p/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7164157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7164709Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:05:07.7165262Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:05:07.7165805Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:05:07.7166400Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:05:07.7166945Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:05:07.7167497Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:05:07.7167766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeh8ccy91
2022-11-23T02:05:07.7168042Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeh8ccy91/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7168278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7168827Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:05:07.7169373Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:05:07.7169916Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:05:07.7170503Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:05:07.7171060Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:05:07.7171599Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:05:07.7171684Z ok (4.024s)
2022-11-23T02:05:07.7171726Z 
2022-11-23T02:05:07.7171988Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7172103Z Ran 1 test in 4.024s
2022-11-23T02:05:07.7172170Z 
2022-11-23T02:05:07.7172267Z OK
2022-11-23T02:05:07.7172286Z 
2022-11-23T02:05:07.7172418Z Generating XML reports...
2022-11-23T02:05:07.7172898Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015731.xml
2022-11-23T02:05:07.7173277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7173458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7173842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7174016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7174279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv480j3nu
2022-11-23T02:05:07.7174560Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv480j3nu/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7174583Z 
2022-11-23T02:05:07.7174693Z Running tests...
2022-11-23T02:05:07.7174963Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7175278Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7175559Z   test_save_load_checkpoint (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7175786Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14400
2022-11-23T02:05:07.7175987Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14401
2022-11-23T02:05:07.7176366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7176543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7177191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7177397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7177778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7177958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7178335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7178531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7178773Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi3r8v_e8
2022-11-23T02:05:07.7179048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi3r8v_e8/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7179389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcajzml3r
2022-11-23T02:05:07.7179685Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcajzml3r/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7179920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7180153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7180407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:05:07.7180654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:05:07.7181044Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.7181450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:05:07.7181693Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.7182001Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.7182242Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.7182479Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.7182586Z ok (7.385s)
2022-11-23T02:05:07.7182606Z 
2022-11-23T02:05:07.7182881Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7182997Z Ran 1 test in 7.385s
2022-11-23T02:05:07.7183017Z 
2022-11-23T02:05:07.7183090Z OK
2022-11-23T02:05:07.7183108Z 
2022-11-23T02:05:07.7183237Z Generating XML reports...
2022-11-23T02:05:07.7183708Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015737.xml
2022-11-23T02:05:07.7184089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7184276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7184662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7184862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7185126Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxetnuzb5
2022-11-23T02:05:07.7185406Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxetnuzb5/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7185426Z 
2022-11-23T02:05:07.7185516Z Running tests...
2022-11-23T02:05:07.7185789Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7186105Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7186384Z   test_sparse_gradients (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7186611Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14616
2022-11-23T02:05:07.7186834Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14617
2022-11-23T02:05:07.7187209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7187393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7187755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7187949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7188324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7188569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7188961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7189161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7189424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptk9fne3r
2022-11-23T02:05:07.7189701Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptk9fne3r/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7189962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp58ojjvgk
2022-11-23T02:05:07.7190217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp58ojjvgk/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7190451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7190685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7190795Z ok (4.257s)
2022-11-23T02:05:07.7190814Z 
2022-11-23T02:05:07.7191088Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7191256Z Ran 1 test in 4.258s
2022-11-23T02:05:07.7191276Z 
2022-11-23T02:05:07.7191371Z OK
2022-11-23T02:05:07.7191389Z 
2022-11-23T02:05:07.7191516Z Generating XML reports...
2022-11-23T02:05:07.7191972Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015747.xml
2022-11-23T02:05:07.7192351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7192531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7192916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7193109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7193373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9zu_nz_z
2022-11-23T02:05:07.7193648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9zu_nz_z/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7193669Z 
2022-11-23T02:05:07.7193781Z Running tests...
2022-11-23T02:05:07.7194052Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7194349Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7194638Z   test_sparse_gradients_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7194861Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14891
2022-11-23T02:05:07.7195085Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14892
2022-11-23T02:05:07.7195463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7195643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7196030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7196224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7196567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7196741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7197123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7197318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7197578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1i730nn2
2022-11-23T02:05:07.7197905Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1i730nn2/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7198171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn2_84tzq
2022-11-23T02:05:07.7198446Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn2_84tzq/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7198684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7198892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7198996Z ok (4.144s)
2022-11-23T02:05:07.7199015Z 
2022-11-23T02:05:07.7199292Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7199407Z Ran 1 test in 4.144s
2022-11-23T02:05:07.7199426Z 
2022-11-23T02:05:07.7199526Z OK
2022-11-23T02:05:07.7199545Z 
2022-11-23T02:05:07.7199670Z Generating XML reports...
2022-11-23T02:05:07.7200151Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015754.xml
2022-11-23T02:05:07.7200527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7200740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7201176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7201398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7201660Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpotslxnwq
2022-11-23T02:05:07.7201938Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpotslxnwq/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7201959Z 
2022-11-23T02:05:07.7202073Z Running tests...
2022-11-23T02:05:07.7202351Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7202673Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7202959Z   test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7203167Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15166
2022-11-23T02:05:07.7203397Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15167
2022-11-23T02:05:07.7203776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7203955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7204337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7204536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7204906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7205087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7205447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7205638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7205902Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoxxfw0je
2022-11-23T02:05:07.7206179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoxxfw0je/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7206417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7206674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_1ws4oc5
2022-11-23T02:05:07.7206948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_1ws4oc5/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7207239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7207493Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.7207714Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.7207819Z ok (7.223s)
2022-11-23T02:05:07.7207839Z 
2022-11-23T02:05:07.7208115Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7208232Z Ran 1 test in 7.223s
2022-11-23T02:05:07.7208252Z 
2022-11-23T02:05:07.7208352Z OK
2022-11-23T02:05:07.7208371Z 
2022-11-23T02:05:07.7208500Z Generating XML reports...
2022-11-23T02:05:07.7208975Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015801.xml
2022-11-23T02:05:07.7209352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7209514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7209979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7210175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7210439Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp347ht33w
2022-11-23T02:05:07.7210714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp347ht33w/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7210734Z 
2022-11-23T02:05:07.7210848Z Running tests...
2022-11-23T02:05:07.7211119Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7211434Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7211724Z   test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7211930Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15381
2022-11-23T02:05:07.7212157Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15382
2022-11-23T02:05:07.7212535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7212719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7213103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7213298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7213661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7213842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7214207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7214407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7214669Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiphmq3d8
2022-11-23T02:05:07.7214946Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiphmq3d8/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7215177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7215439Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt4knjj9h
2022-11-23T02:05:07.7215714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt4knjj9h/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7215948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7216246Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.7216475Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:05:07.7216584Z ok (6.565s)
2022-11-23T02:05:07.7216604Z 
2022-11-23T02:05:07.7217127Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7217249Z Ran 1 test in 6.565s
2022-11-23T02:05:07.7217269Z 
2022-11-23T02:05:07.7217365Z OK
2022-11-23T02:05:07.7217384Z 
2022-11-23T02:05:07.7217513Z Generating XML reports...
2022-11-23T02:05:07.7217995Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015810.xml
2022-11-23T02:05:07.7218374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7218533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7218923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7219118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7219459Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph75z3mqj
2022-11-23T02:05:07.7219735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph75z3mqj/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7219756Z 
2022-11-23T02:05:07.7219871Z Running tests...
2022-11-23T02:05:07.7220140Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7220456Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7220803Z   test_allgather_coalesced (__main__.GlooProcessGroupWithDispatchedCollectivesTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7221005Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15596
2022-11-23T02:05:07.7221388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7221568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7221951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7222142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7222401Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcy0zfwyw
2022-11-23T02:05:07.7222678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcy0zfwyw/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7222913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7223141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:05:07.7223553Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:05:07.7224323Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:05:07.7224446Z   warnings.warn(
2022-11-23T02:05:07.7224551Z ok (4.039s)
2022-11-23T02:05:07.7224571Z 
2022-11-23T02:05:07.7224838Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7224959Z Ran 1 test in 4.039s
2022-11-23T02:05:07.7224979Z 
2022-11-23T02:05:07.7225071Z OK
2022-11-23T02:05:07.7225089Z 
2022-11-23T02:05:07.7225216Z Generating XML reports...
2022-11-23T02:05:07.7225763Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20221123015819.xml
2022-11-23T02:05:07.7226218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7226409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7226803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7226999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7227263Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz5f10ti4
2022-11-23T02:05:07.7227540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz5f10ti4/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7227561Z 
2022-11-23T02:05:07.7227675Z Running tests...
2022-11-23T02:05:07.7227941Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7228240Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7228585Z   test_allreduce_coalesced (__main__.GlooProcessGroupWithDispatchedCollectivesTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7228816Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15732
2022-11-23T02:05:07.7229252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7229433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7229818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7230017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7230279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2hgqbqv9
2022-11-23T02:05:07.7230534Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2hgqbqv9/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7230773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7231025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:05:07.7231440Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:05:07.7232185Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:05:07.7232305Z   warnings.warn(
2022-11-23T02:05:07.7232410Z ok (4.109s)
2022-11-23T02:05:07.7232429Z 
2022-11-23T02:05:07.7232698Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7232815Z Ran 1 test in 4.110s
2022-11-23T02:05:07.7232834Z 
2022-11-23T02:05:07.7232908Z OK
2022-11-23T02:05:07.7232949Z 
2022-11-23T02:05:07.7233060Z Generating XML reports...
2022-11-23T02:05:07.7233630Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20221123015826.xml
2022-11-23T02:05:07.7234011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7234192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7234578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7234774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7235036Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwdti3q_b
2022-11-23T02:05:07.7235315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwdti3q_b/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7235335Z 
2022-11-23T02:05:07.7235425Z Running tests...
2022-11-23T02:05:07.7235750Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7236087Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7236420Z   test_collectives (__main__.GlooProcessGroupWithDispatchedCollectivesTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7236640Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15868
2022-11-23T02:05:07.7237012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7237234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7237678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7237874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7238116Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj_hm36wa
2022-11-23T02:05:07.7238390Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj_hm36wa/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7238678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7238927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:05:07.7239337Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:05:07.7239446Z ok (4.072s)
2022-11-23T02:05:07.7239466Z 
2022-11-23T02:05:07.7239732Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7239850Z Ran 1 test in 4.072s
2022-11-23T02:05:07.7239870Z 
2022-11-23T02:05:07.7239943Z OK
2022-11-23T02:05:07.7239987Z 
2022-11-23T02:05:07.7240164Z Generating XML reports...
2022-11-23T02:05:07.7240762Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20221123015832.xml
2022-11-23T02:05:07.7241144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7241327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7241711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7241906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7242171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6kfgwuvj
2022-11-23T02:05:07.7242448Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6kfgwuvj/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7242468Z 
2022-11-23T02:05:07.7242558Z Running tests...
2022-11-23T02:05:07.7242830Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7243151Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7243491Z   test_monitored_barrier (__main__.GlooProcessGroupWithDispatchedCollectivesTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7243716Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16004
2022-11-23T02:05:07.7244094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7244271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7244655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7244829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7245091Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqytac59s
2022-11-23T02:05:07.7245462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqytac59s/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7245709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7245963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:05:07.7246373Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:05:07.7246479Z ok (4.047s)
2022-11-23T02:05:07.7246498Z 
2022-11-23T02:05:07.7246763Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7246879Z Ran 1 test in 4.047s
2022-11-23T02:05:07.7246899Z 
2022-11-23T02:05:07.7246972Z OK
2022-11-23T02:05:07.7246991Z 
2022-11-23T02:05:07.7247119Z Generating XML reports...
2022-11-23T02:05:07.7247680Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20221123015839.xml
2022-11-23T02:05:07.7248053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7248290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7248680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7248871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7249134Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi4pgvfgq
2022-11-23T02:05:07.7249413Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi4pgvfgq/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7249433Z 
2022-11-23T02:05:07.7249523Z Running tests...
2022-11-23T02:05:07.7249794Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7250112Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7250367Z   test_allgather_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7250594Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16140
2022-11-23T02:05:07.7250821Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16141
2022-11-23T02:05:07.7251083Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16142
2022-11-23T02:05:07.7251316Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16143
2022-11-23T02:05:07.7251672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7251850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7252234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7252438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7252805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7252985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7253364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7253561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7253932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7254088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7254469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7254661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7255090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7255277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7255655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7255848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7256111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqlvju3gf
2022-11-23T02:05:07.7256367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqlvju3gf/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7256627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ynb3g0a
2022-11-23T02:05:07.7257169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ynb3g0a/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7257418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7257732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7257989Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt8ld19se
2022-11-23T02:05:07.7258262Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt8ld19se/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7258498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7258763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7t7pxr2y
2022-11-23T02:05:07.7259016Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7t7pxr2y/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7259247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7259351Z ok (4.249s)
2022-11-23T02:05:07.7259371Z 
2022-11-23T02:05:07.7259658Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7259776Z Ran 1 test in 4.249s
2022-11-23T02:05:07.7259795Z 
2022-11-23T02:05:07.7259893Z OK
2022-11-23T02:05:07.7259912Z 
2022-11-23T02:05:07.7260042Z Generating XML reports...
2022-11-23T02:05:07.7260487Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123015845.xml
2022-11-23T02:05:07.7260841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7261020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7261403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7261602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7261867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgc01ey0d
2022-11-23T02:05:07.7262144Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgc01ey0d/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7262168Z 
2022-11-23T02:05:07.7262278Z Running tests...
2022-11-23T02:05:07.7262551Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7262868Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7263109Z   test_allgather_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7263332Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16483
2022-11-23T02:05:07.7263553Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16484
2022-11-23T02:05:07.7263773Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16485
2022-11-23T02:05:07.7263987Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16486
2022-11-23T02:05:07.7264430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7264624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7265014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7265189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7265560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7265736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7266111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7266307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7266674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7266898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7267274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7267444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7267813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7267988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7268363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7268552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7268819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmqnl2ysn
2022-11-23T02:05:07.7269097Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmqnl2ysn/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7269336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7269594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_gd30gfw
2022-11-23T02:05:07.7269843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_gd30gfw/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7270106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpksn6d9lp
2022-11-23T02:05:07.7270381Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpksn6d9lp/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7270619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7270853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7271116Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptfcv_k7y
2022-11-23T02:05:07.7271390Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptfcv_k7y/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7271618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7271704Z ok (6.081s)
2022-11-23T02:05:07.7271745Z 
2022-11-23T02:05:07.7272000Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7272120Z Ran 1 test in 6.082s
2022-11-23T02:05:07.7272140Z 
2022-11-23T02:05:07.7272241Z OK
2022-11-23T02:05:07.7272261Z 
2022-11-23T02:05:07.7272392Z Generating XML reports...
2022-11-23T02:05:07.7272835Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123015852.xml
2022-11-23T02:05:07.7273261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7273492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7273897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7274070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7274330Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3zk3t0vf
2022-11-23T02:05:07.7274607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3zk3t0vf/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7274627Z 
2022-11-23T02:05:07.7274742Z Running tests...
2022-11-23T02:05:07.7275013Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7275329Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7275589Z   test_allgather_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7275820Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16830
2022-11-23T02:05:07.7276073Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16831
2022-11-23T02:05:07.7276291Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16832
2022-11-23T02:05:07.7276506Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16833
2022-11-23T02:05:07.7276893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7277073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7277455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7277651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7278020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7278196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7278558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7278752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7279122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7279297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7279675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7279868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7280231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7280413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7280769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7280960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7281225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp29c7to7h
2022-11-23T02:05:07.7281498Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp29c7to7h/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7281759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbx4p4ypp
2022-11-23T02:05:07.7282037Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbx4p4ypp/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7282271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7282561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7282832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6s5mk474
2022-11-23T02:05:07.7283083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6s5mk474/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7283316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7283578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg151xwqe
2022-11-23T02:05:07.7283850Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg151xwqe/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7284081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7284186Z ok (4.260s)
2022-11-23T02:05:07.7284206Z 
2022-11-23T02:05:07.7284480Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7284599Z Ran 1 test in 4.260s
2022-11-23T02:05:07.7284621Z 
2022-11-23T02:05:07.7284695Z OK
2022-11-23T02:05:07.7284736Z 
2022-11-23T02:05:07.7284896Z Generating XML reports...
2022-11-23T02:05:07.7285342Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123015901.xml
2022-11-23T02:05:07.7285720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7285905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7286287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7286484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7286746Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpum2of_c4
2022-11-23T02:05:07.7287024Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpum2of_c4/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7287044Z 
2022-11-23T02:05:07.7287134Z Running tests...
2022-11-23T02:05:07.7287407Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7287726Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7287999Z   test_allgather_coalesced_async (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7288223Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17173
2022-11-23T02:05:07.7288448Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17174
2022-11-23T02:05:07.7288671Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17175
2022-11-23T02:05:07.7288889Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17176
2022-11-23T02:05:07.7289242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7289426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7289814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7290012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7290382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7290559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7290934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7291127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7291494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7291702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7292104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7292302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7292669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7292844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7293223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7293417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7293685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8wp34yus
2022-11-23T02:05:07.7293945Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8wp34yus/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7294210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2zssexm5
2022-11-23T02:05:07.7294540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2zssexm5/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7294800Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjpt9jpsj
2022-11-23T02:05:07.7295074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjpt9jpsj/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7295310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7295546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7295779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7296038Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpji95my0_
2022-11-23T02:05:07.7296290Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpji95my0_/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7296521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7297016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:05:07.7297275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:05:07.7297522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:05:07.7297763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:05:07.7298182Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:05:07.7298589Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:05:07.7298991Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:05:07.7299379Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:05:07.7300138Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:05:07.7300255Z   warnings.warn(
2022-11-23T02:05:07.7300998Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:05:07.7301192Z   warnings.warn(
2022-11-23T02:05:07.7301944Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:05:07.7302061Z   warnings.warn(
2022-11-23T02:05:07.7302787Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:05:07.7302901Z   warnings.warn(
2022-11-23T02:05:07.7303004Z ok (4.283s)
2022-11-23T02:05:07.7303024Z 
2022-11-23T02:05:07.7303297Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7303392Z Ran 1 test in 4.283s
2022-11-23T02:05:07.7303411Z 
2022-11-23T02:05:07.7303514Z OK
2022-11-23T02:05:07.7303533Z 
2022-11-23T02:05:07.7303670Z Generating XML reports...
2022-11-23T02:05:07.7304187Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123015908.xml
2022-11-23T02:05:07.7304559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7304735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7305117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7305309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7305547Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpigg0u74t
2022-11-23T02:05:07.7305822Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpigg0u74t/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7305842Z 
2022-11-23T02:05:07.7305954Z Running tests...
2022-11-23T02:05:07.7306224Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7306544Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7306816Z   test_allgather_coalesced_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7307043Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17516
2022-11-23T02:05:07.7307268Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17517
2022-11-23T02:05:07.7307468Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17518
2022-11-23T02:05:07.7307684Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17519
2022-11-23T02:05:07.7308068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7308253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7308641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7308839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7309211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7309390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7309769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7309940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7310314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7310494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7310924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7311127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7311497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7311673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7312058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7312251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7312493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy0esy1kh
2022-11-23T02:05:07.7312770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy0esy1kh/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7313033Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq9mehb5l
2022-11-23T02:05:07.7313362Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq9mehb5l/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7313601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7313859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4stjl7un
2022-11-23T02:05:07.7314129Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4stjl7un/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7314386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjb9y9745
2022-11-23T02:05:07.7314596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7314869Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjb9y9745/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7315101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7315335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7316091Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:05:07.7316214Z   warnings.warn(
2022-11-23T02:05:07.7316956Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:05:07.7317069Z   warnings.warn(
2022-11-23T02:05:07.7317806Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:05:07.7317926Z   warnings.warn(
2022-11-23T02:05:07.7318652Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:05:07.7318744Z   warnings.warn(
2022-11-23T02:05:07.7318847Z ok (4.247s)
2022-11-23T02:05:07.7318866Z 
2022-11-23T02:05:07.7319135Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7319250Z Ran 1 test in 4.247s
2022-11-23T02:05:07.7319270Z 
2022-11-23T02:05:07.7319362Z OK
2022-11-23T02:05:07.7319380Z 
2022-11-23T02:05:07.7319507Z Generating XML reports...
2022-11-23T02:05:07.7320008Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123015914.xml
2022-11-23T02:05:07.7320400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7320559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7320938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7321130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7321393Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1mwh88y1
2022-11-23T02:05:07.7321665Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1mwh88y1/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7321685Z 
2022-11-23T02:05:07.7321793Z Running tests...
2022-11-23T02:05:07.7322058Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7322377Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7322709Z   test_allgather_noncontiguous_input (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7322914Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17859
2022-11-23T02:05:07.7323132Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17860
2022-11-23T02:05:07.7323345Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17861
2022-11-23T02:05:07.7323559Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17862
2022-11-23T02:05:07.7323938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7324116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7324505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7324698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7325053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7325226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7325596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7325784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7326145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7326319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7326691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7326877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7327238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7327393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7327775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7327960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7328216Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkxojzjac
2022-11-23T02:05:07.7328489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkxojzjac/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7328719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7329028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpolsnjzyw
2022-11-23T02:05:07.7329307Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpolsnjzyw/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7329520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7329781Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppt47mwnw
2022-11-23T02:05:07.7330051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppt47mwnw/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7330310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpddzc0o5_
2022-11-23T02:05:07.7330575Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpddzc0o5_/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7330805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7331035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7331204Z ok (4.229s)
2022-11-23T02:05:07.7331225Z 
2022-11-23T02:05:07.7331554Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7331652Z Ran 1 test in 4.229s
2022-11-23T02:05:07.7331672Z 
2022-11-23T02:05:07.7331764Z OK
2022-11-23T02:05:07.7331782Z 
2022-11-23T02:05:07.7331906Z Generating XML reports...
2022-11-23T02:05:07.7332343Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123015921.xml
2022-11-23T02:05:07.7332714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7332889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7333270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7333466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7333705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfy9y6w22
2022-11-23T02:05:07.7333977Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfy9y6w22/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7333997Z 
2022-11-23T02:05:07.7334106Z Running tests...
2022-11-23T02:05:07.7334371Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7334683Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7334933Z   test_allgather_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7335154Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18202
2022-11-23T02:05:07.7335375Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18203
2022-11-23T02:05:07.7335573Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18204
2022-11-23T02:05:07.7335792Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18205
2022-11-23T02:05:07.7336170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7336346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7336976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7337182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7337559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7337736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7338108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7338359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7338743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7338922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7339297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7339490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7339857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7340029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7340505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7340683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7340945Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjzinmipq
2022-11-23T02:05:07.7341297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjzinmipq/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7341556Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg_uyavgb
2022-11-23T02:05:07.7341825Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg_uyavgb/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7342081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaqnvws4l
2022-11-23T02:05:07.7342350Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaqnvws4l/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7342580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7342833Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptnxbn_t5
2022-11-23T02:05:07.7343084Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptnxbn_t5/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7343317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7343540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7343763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7343865Z ok (4.592s)
2022-11-23T02:05:07.7343884Z 
2022-11-23T02:05:07.7344164Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7344276Z Ran 1 test in 4.592s
2022-11-23T02:05:07.7344296Z 
2022-11-23T02:05:07.7344387Z OK
2022-11-23T02:05:07.7344406Z 
2022-11-23T02:05:07.7344510Z Generating XML reports...
2022-11-23T02:05:07.7344951Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123015928.xml
2022-11-23T02:05:07.7345327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7345508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7345890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7346080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7346340Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpldm4n8fo
2022-11-23T02:05:07.7346610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpldm4n8fo/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7346631Z 
2022-11-23T02:05:07.7346740Z Running tests...
2022-11-23T02:05:07.7346989Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7347303Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7347613Z   test_allgather_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7347844Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18569
2022-11-23T02:05:07.7348062Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18570
2022-11-23T02:05:07.7348280Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18571
2022-11-23T02:05:07.7348493Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18572
2022-11-23T02:05:07.7348872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7349031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7349410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7349604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7349967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7350193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7350571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7350763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7351127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7351301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7351656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7351843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7352215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7352388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7352754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7352940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7353201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp53qvpt11
2022-11-23T02:05:07.7353475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp53qvpt11/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7353688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7353945Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo8n106ab
2022-11-23T02:05:07.7354217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo8n106ab/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7354445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7354699Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9h2eo0cn
2022-11-23T02:05:07.7354969Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9h2eo0cn/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7355195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7355449Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfa0c90wm
2022-11-23T02:05:07.7355716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfa0c90wm/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7355923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7356023Z ok (7.453s)
2022-11-23T02:05:07.7356043Z 
2022-11-23T02:05:07.7356362Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7356480Z Ran 1 test in 7.453s
2022-11-23T02:05:07.7356502Z 
2022-11-23T02:05:07.7356593Z OK
2022-11-23T02:05:07.7356612Z 
2022-11-23T02:05:07.7356735Z Generating XML reports...
2022-11-23T02:05:07.7357175Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123015935.xml
2022-11-23T02:05:07.7357544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7357705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7358092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7358285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7358540Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpernd6yge
2022-11-23T02:05:07.7358813Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpernd6yge/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7358883Z 
2022-11-23T02:05:07.7358997Z Running tests...
2022-11-23T02:05:07.7359264Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7359576Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7359829Z   test_allreduce_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7360031Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18940
2022-11-23T02:05:07.7360249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18941
2022-11-23T02:05:07.7360465Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18942
2022-11-23T02:05:07.7360679Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18943
2022-11-23T02:05:07.7361055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7361231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7361611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7361804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7362152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7362326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7362700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7362892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7363257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7363430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7363804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7363991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7364350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7364506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7364884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7365072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7365329Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkv3_we_r
2022-11-23T02:05:07.7365650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkv3_we_r/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7365917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp32ddbaxx
2022-11-23T02:05:07.7366189Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp32ddbaxx/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7366443Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb8hrtxmy
2022-11-23T02:05:07.7366693Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb8hrtxmy/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7366922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7367150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7367377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7367630Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpveojsfjz
2022-11-23T02:05:07.7367952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpveojsfjz/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7368178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7368279Z ok (4.251s)
2022-11-23T02:05:07.7368298Z 
2022-11-23T02:05:07.7368570Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7368664Z Ran 1 test in 4.251s
2022-11-23T02:05:07.7368683Z 
2022-11-23T02:05:07.7368775Z OK
2022-11-23T02:05:07.7368794Z 
2022-11-23T02:05:07.7368922Z Generating XML reports...
2022-11-23T02:05:07.7369359Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123015945.xml
2022-11-23T02:05:07.7369731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7369909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7370288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7370485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7370723Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph7w424yh
2022-11-23T02:05:07.7370995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph7w424yh/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7371017Z 
2022-11-23T02:05:07.7371125Z Running tests...
2022-11-23T02:05:07.7371388Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7371704Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7371962Z   test_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7372186Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19283
2022-11-23T02:05:07.7372492Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19284
2022-11-23T02:05:07.7372713Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19285
2022-11-23T02:05:07.7372911Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19286
2022-11-23T02:05:07.7373293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7373468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7373848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7374039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7374463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7374645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7375024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7375195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7375556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7375728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7376106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7376298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7376664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7377088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7377479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7377755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7378000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpei8qyn_u
2022-11-23T02:05:07.7378277Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpei8qyn_u/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7378508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7378764Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphd_r33kb
2022-11-23T02:05:07.7379032Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphd_r33kb/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7379267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7379522Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplkwfvu2j
2022-11-23T02:05:07.7379796Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplkwfvu2j/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7380031Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpck12qgnb
2022-11-23T02:05:07.7380297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpck12qgnb/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7380524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7380749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7380850Z ok (6.144s)
2022-11-23T02:05:07.7380870Z 
2022-11-23T02:05:07.7381145Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7381261Z Ran 1 test in 6.145s
2022-11-23T02:05:07.7381281Z 
2022-11-23T02:05:07.7381375Z OK
2022-11-23T02:05:07.7381394Z 
2022-11-23T02:05:07.7381518Z Generating XML reports...
2022-11-23T02:05:07.7381944Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123015951.xml
2022-11-23T02:05:07.7382312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7382487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7382868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7383060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7383315Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbmw7_3jh
2022-11-23T02:05:07.7383583Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbmw7_3jh/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7383688Z 
2022-11-23T02:05:07.7383805Z Running tests...
2022-11-23T02:05:07.7384061Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7384383Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7384663Z   test_allreduce_basics_cuda_using_work_api (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7384884Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19630
2022-11-23T02:05:07.7385101Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19631
2022-11-23T02:05:07.7385318Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19632
2022-11-23T02:05:07.7385528Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19633
2022-11-23T02:05:07.7385902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7386079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7386441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7386729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7387098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7387269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7387638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7387827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7388188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7388363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7388714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7388906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7389282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7389454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7389819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7390004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7390264Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpegsg89tr
2022-11-23T02:05:07.7390537Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpegsg89tr/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7390771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7391012Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp60sw105c
2022-11-23T02:05:07.7391280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp60sw105c/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7391533Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsv54ojyz
2022-11-23T02:05:07.7391799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsv54ojyz/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7392028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7392282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1oou131w
2022-11-23T02:05:07.7392550Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1oou131w/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7392843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7393061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7393162Z ok (6.056s)
2022-11-23T02:05:07.7393182Z 
2022-11-23T02:05:07.7393453Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7393564Z Ran 1 test in 6.057s
2022-11-23T02:05:07.7393584Z 
2022-11-23T02:05:07.7393676Z OK
2022-11-23T02:05:07.7393695Z 
2022-11-23T02:05:07.7393819Z Generating XML reports...
2022-11-23T02:05:07.7394257Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020000.xml
2022-11-23T02:05:07.7394626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7394800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7395165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7395421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7395678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm30rm043
2022-11-23T02:05:07.7395946Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm30rm043/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7395966Z 
2022-11-23T02:05:07.7396073Z Running tests...
2022-11-23T02:05:07.7396339Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7396653Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7396925Z   test_allreduce_basics_using_work_api (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7397127Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19977
2022-11-23T02:05:07.7397349Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19978
2022-11-23T02:05:07.7397565Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19979
2022-11-23T02:05:07.7397785Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19980
2022-11-23T02:05:07.7398161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7398336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7398715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7398906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7399270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7399427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7399810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7400003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7400363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7400536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7400910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7401099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7401468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7401625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7402045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7402243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7402505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpumctsdib
2022-11-23T02:05:07.7402781Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpumctsdib/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7403011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7403269Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqd5jg25e
2022-11-23T02:05:07.7403541Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqd5jg25e/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7403795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_19gpkqw
2022-11-23T02:05:07.7404048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_19gpkqw/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7404275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7404557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7404816Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsdyn88e5
2022-11-23T02:05:07.7405083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsdyn88e5/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7405308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7405408Z ok (4.240s)
2022-11-23T02:05:07.7405428Z 
2022-11-23T02:05:07.7405700Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7405794Z Ran 1 test in 4.240s
2022-11-23T02:05:07.7405813Z 
2022-11-23T02:05:07.7405904Z OK
2022-11-23T02:05:07.7405922Z 
2022-11-23T02:05:07.7406045Z Generating XML reports...
2022-11-23T02:05:07.7406486Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020009.xml
2022-11-23T02:05:07.7406857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7407032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7407415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7407605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7407860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6esam2ju
2022-11-23T02:05:07.7408111Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6esam2ju/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7408134Z 
2022-11-23T02:05:07.7408243Z Running tests...
2022-11-23T02:05:07.7408511Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7408827Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7409082Z   test_allreduce_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7409303Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20320
2022-11-23T02:05:07.7409520Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20321
2022-11-23T02:05:07.7409736Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20322
2022-11-23T02:05:07.7409931Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20323
2022-11-23T02:05:07.7410428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7410604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7411034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7411235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7411601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7411774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7412148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7412338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7412679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7412852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7413226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7413413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7413836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7414015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7414394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7414584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7414825Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5fri0i2y
2022-11-23T02:05:07.7415098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5fri0i2y/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7415355Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp695pg_4g
2022-11-23T02:05:07.7415626Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp695pg_4g/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7415857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7416115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvem16fnh
2022-11-23T02:05:07.7416383Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvem16fnh/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7416612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7417114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp94nxycwy
2022-11-23T02:05:07.7417378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp94nxycwy/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7417607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7417833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7417933Z ok (4.229s)
2022-11-23T02:05:07.7417957Z 
2022-11-23T02:05:07.7418235Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7418347Z Ran 1 test in 4.229s
2022-11-23T02:05:07.7418367Z 
2022-11-23T02:05:07.7418464Z OK
2022-11-23T02:05:07.7418483Z 
2022-11-23T02:05:07.7418608Z Generating XML reports...
2022-11-23T02:05:07.7419028Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020015.xml
2022-11-23T02:05:07.7419399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7419574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7419956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7420221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7420488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgorgl_uv
2022-11-23T02:05:07.7420761Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgorgl_uv/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7420781Z 
2022-11-23T02:05:07.7420889Z Running tests...
2022-11-23T02:05:07.7421159Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7421458Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7421727Z   test_allreduce_coalesced_async (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7421947Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20663
2022-11-23T02:05:07.7422165Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20664
2022-11-23T02:05:07.7422385Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20665
2022-11-23T02:05:07.7422598Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20666
2022-11-23T02:05:07.7423050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7423227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7423591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7423781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7424146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7424320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7424700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7424888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7425254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7425426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7425806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7425975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7426338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7426509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7426884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7427077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7427336Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn0rjt_bk
2022-11-23T02:05:07.7427610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn0rjt_bk/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7427839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7428079Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3sw1iu9c
2022-11-23T02:05:07.7428349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3sw1iu9c/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7428604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp96c8hs_k
2022-11-23T02:05:07.7428871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp96c8hs_k/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7429158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7429393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7429653Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6y4tfr55
2022-11-23T02:05:07.7429921Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6y4tfr55/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7430148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7430376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:05:07.7430619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:05:07.7430860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:05:07.7431101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:05:07.7431514Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:05:07.7431968Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:05:07.7432355Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:05:07.7432753Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:05:07.7433502Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:05:07.7433617Z   warnings.warn(
2022-11-23T02:05:07.7434345Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:05:07.7434459Z   warnings.warn(
2022-11-23T02:05:07.7435197Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:05:07.7435309Z   warnings.warn(
2022-11-23T02:05:07.7436037Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:05:07.7436149Z   warnings.warn(
2022-11-23T02:05:07.7436247Z ok (4.230s)
2022-11-23T02:05:07.7436267Z 
2022-11-23T02:05:07.7436539Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7436650Z Ran 1 test in 4.231s
2022-11-23T02:05:07.7436670Z 
2022-11-23T02:05:07.7436742Z OK
2022-11-23T02:05:07.7436777Z 
2022-11-23T02:05:07.7436884Z Generating XML reports...
2022-11-23T02:05:07.7437322Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020022.xml
2022-11-23T02:05:07.7437694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7437871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7438249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7438493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7438756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoe1ei8ma
2022-11-23T02:05:07.7439031Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoe1ei8ma/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7439051Z 
2022-11-23T02:05:07.7439140Z Running tests...
2022-11-23T02:05:07.7439409Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7439719Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7439986Z   test_allreduce_coalesced_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7440292Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21006
2022-11-23T02:05:07.7440517Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21007
2022-11-23T02:05:07.7440734Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21008
2022-11-23T02:05:07.7440953Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21009
2022-11-23T02:05:07.7441378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7441555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7441935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7442128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7442491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7442666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7443038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7443230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7443590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7443745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7444125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7444312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7444676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7444848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7445221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7445413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7445673Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps3ay9gko
2022-11-23T02:05:07.7445930Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps3ay9gko/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7446160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7446417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoeazmbev
2022-11-23T02:05:07.7446688Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoeazmbev/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7446942Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxi22a2wu
2022-11-23T02:05:07.7447210Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxi22a2wu/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7447437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7447726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7447992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjo77a26t
2022-11-23T02:05:07.7448239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjo77a26t/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7448467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7448568Z ok (4.237s)
2022-11-23T02:05:07.7448587Z 
2022-11-23T02:05:07.7448858Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7448969Z Ran 1 test in 4.237s
2022-11-23T02:05:07.7448988Z 
2022-11-23T02:05:07.7449079Z OK
2022-11-23T02:05:07.7449098Z 
2022-11-23T02:05:07.7449223Z Generating XML reports...
2022-11-23T02:05:07.7449660Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020029.xml
2022-11-23T02:05:07.7450014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7450243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7450625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7450817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7451077Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx1w3ad_f
2022-11-23T02:05:07.7451347Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx1w3ad_f/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7451368Z 
2022-11-23T02:05:07.7451475Z Running tests...
2022-11-23T02:05:07.7451740Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7452052Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7452306Z   test_allreduce_coalesced_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7452529Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21349
2022-11-23T02:05:07.7452748Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21350
2022-11-23T02:05:07.7452971Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21351
2022-11-23T02:05:07.7453185Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21352
2022-11-23T02:05:07.7453559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7453736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7454114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7454291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7454654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7454829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7455204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7455397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7455757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7455930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7456299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7456488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7457174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7457372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7457758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7457951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7458213Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdyti89dg
2022-11-23T02:05:07.7458487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdyti89dg/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7458717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7458973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvjigmlzg
2022-11-23T02:05:07.7459230Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvjigmlzg/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7459465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7459791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph_w34h82
2022-11-23T02:05:07.7460057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph_w34h82/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7460309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnpfssnma
2022-11-23T02:05:07.7460581Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnpfssnma/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7460807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7461032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7461132Z ok (4.247s)
2022-11-23T02:05:07.7461152Z 
2022-11-23T02:05:07.7461414Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7461529Z Ran 1 test in 4.247s
2022-11-23T02:05:07.7461550Z 
2022-11-23T02:05:07.7461643Z OK
2022-11-23T02:05:07.7461662Z 
2022-11-23T02:05:07.7461786Z Generating XML reports...
2022-11-23T02:05:07.7462227Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020035.xml
2022-11-23T02:05:07.7462600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7462776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7463148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7463345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7463613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmavcdq0n
2022-11-23T02:05:07.7463888Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmavcdq0n/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7463912Z 
2022-11-23T02:05:07.7464018Z Running tests...
2022-11-23T02:05:07.7464287Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7464582Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7464861Z   test_allreduce_coalesced_checks_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7465079Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21692
2022-11-23T02:05:07.7465296Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21693
2022-11-23T02:05:07.7465510Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21694
2022-11-23T02:05:07.7465719Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21695
2022-11-23T02:05:07.7466143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7466329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7466714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7466887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7467250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7467421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7467784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7467957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7468336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7468580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7468961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7469129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7469495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7469668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7470036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7470224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7470486Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl6oq0p5a
2022-11-23T02:05:07.7470757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl6oq0p5a/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7471016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3pa94ena
2022-11-23T02:05:07.7471285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3pa94ena/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7471498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7471750Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm6b41ttp
2022-11-23T02:05:07.7472016Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm6b41ttp/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7472271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmtnnj1c_
2022-11-23T02:05:07.7472537Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmtnnj1c_/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7472867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7473095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7473321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7473403Z ok (6.085s)
2022-11-23T02:05:07.7473439Z 
2022-11-23T02:05:07.7473691Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7473800Z Ran 1 test in 6.085s
2022-11-23T02:05:07.7473820Z 
2022-11-23T02:05:07.7473910Z OK
2022-11-23T02:05:07.7473928Z 
2022-11-23T02:05:07.7474053Z Generating XML reports...
2022-11-23T02:05:07.7474489Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020042.xml
2022-11-23T02:05:07.7474861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7475087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7475477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7475650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7475905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf_2flbe2
2022-11-23T02:05:07.7476175Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf_2flbe2/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7476195Z 
2022-11-23T02:05:07.7476301Z Running tests...
2022-11-23T02:05:07.7476568Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7476880Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7477146Z   test_allreduce_coalesced_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7477368Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22039
2022-11-23T02:05:07.7477627Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22040
2022-11-23T02:05:07.7477843Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22041
2022-11-23T02:05:07.7479132Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22042
2022-11-23T02:05:07.7479518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7479696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7480075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7480266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7480635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7480811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7481173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7481366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7481731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7481903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7482278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7482466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7482835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7483010Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7483393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7483565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7483822Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmjn753q8
2022-11-23T02:05:07.7484092Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmjn753q8/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7484351Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx83s9iju
2022-11-23T02:05:07.7484621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx83s9iju/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7484850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7485137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7485401Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp21h9e4ta
2022-11-23T02:05:07.7485653Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp21h9e4ta/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7485880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7486137Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpat_134le
2022-11-23T02:05:07.7486403Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpat_134le/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7486632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7486733Z ok (4.616s)
2022-11-23T02:05:07.7486752Z 
2022-11-23T02:05:07.7487032Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7487142Z Ran 1 test in 4.617s
2022-11-23T02:05:07.7487164Z 
2022-11-23T02:05:07.7487257Z OK
2022-11-23T02:05:07.7487275Z 
2022-11-23T02:05:07.7487434Z Generating XML reports...
2022-11-23T02:05:07.7487875Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020051.xml
2022-11-23T02:05:07.7488246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7488422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7488801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7488992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7489248Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8k7am90s
2022-11-23T02:05:07.7489524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8k7am90s/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7489543Z 
2022-11-23T02:05:07.7489653Z Running tests...
2022-11-23T02:05:07.7489906Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7490219Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7490470Z   test_allreduce_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7490689Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22406
2022-11-23T02:05:07.7490907Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22407
2022-11-23T02:05:07.7491124Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22408
2022-11-23T02:05:07.7491336Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22409
2022-11-23T02:05:07.7491709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7491871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7492257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7492448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7492813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7492986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7493365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7493553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7493921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7494126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7494514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7494705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7495068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7495245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7495617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7495804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7496064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjajg70dq
2022-11-23T02:05:07.7496338Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjajg70dq/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7496579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjzqe1ur4
2022-11-23T02:05:07.7497171Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjzqe1ur4/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7497410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7497641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7497899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo6_8j411
2022-11-23T02:05:07.7498166Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo6_8j411/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7498392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7498650Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp78k2gh8r
2022-11-23T02:05:07.7498904Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp78k2gh8r/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7499136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7499236Z ok (4.548s)
2022-11-23T02:05:07.7499258Z 
2022-11-23T02:05:07.7499537Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7499654Z Ran 1 test in 4.548s
2022-11-23T02:05:07.7499674Z 
2022-11-23T02:05:07.7499765Z OK
2022-11-23T02:05:07.7499784Z 
2022-11-23T02:05:07.7499910Z Generating XML reports...
2022-11-23T02:05:07.7500353Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020058.xml
2022-11-23T02:05:07.7500726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7500883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7501354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7501554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7501809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp47mieizr
2022-11-23T02:05:07.7502080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp47mieizr/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7502100Z 
2022-11-23T02:05:07.7502208Z Running tests...
2022-11-23T02:05:07.7502478Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7502794Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7503033Z   test_allreduce_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7503258Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22773
2022-11-23T02:05:07.7503562Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22774
2022-11-23T02:05:07.7503792Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22775
2022-11-23T02:05:07.7504015Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22776
2022-11-23T02:05:07.7504397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7504573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7504953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7505144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7505497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7505670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7506052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7506315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7506684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7506856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7507231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7507420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7507762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7507934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7508310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7508501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7508764Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppmdbh4d7
2022-11-23T02:05:07.7509042Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppmdbh4d7/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7509278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7509535Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfhmk05i7
2022-11-23T02:05:07.7509805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfhmk05i7/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7510042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0mfgq3dm
2022-11-23T02:05:07.7510315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0mfgq3dm/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7510543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7510797Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7_mkgkvi
2022-11-23T02:05:07.7511065Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7_mkgkvi/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7511294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7511520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7511621Z ok (6.348s)
2022-11-23T02:05:07.7511641Z 
2022-11-23T02:05:07.7511897Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7512009Z Ran 1 test in 6.348s
2022-11-23T02:05:07.7512028Z 
2022-11-23T02:05:07.7512120Z OK
2022-11-23T02:05:07.7512139Z 
2022-11-23T02:05:07.7512263Z Generating XML reports...
2022-11-23T02:05:07.7512764Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020105.xml
2022-11-23T02:05:07.7513151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7513327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7513707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7513901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7514139Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzbz83mpl
2022-11-23T02:05:07.7514413Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzbz83mpl/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7514433Z 
2022-11-23T02:05:07.7514542Z Running tests...
2022-11-23T02:05:07.7514813Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7515128Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7515435Z   test_barrier_implies_wait (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7515655Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23144
2022-11-23T02:05:07.7515874Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23145
2022-11-23T02:05:07.7516073Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23146
2022-11-23T02:05:07.7516286Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23147
2022-11-23T02:05:07.7516664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7516839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7517222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7517416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7517780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7517954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7518328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7518500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7518862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7519039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7519410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7519599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7519970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7520145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7520522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7520692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7520945Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ecp2_lz
2022-11-23T02:05:07.7521216Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ecp2_lz/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7521447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7521803Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq28_v6cg
2022-11-23T02:05:07.7522083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq28_v6cg/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7522312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7522567Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0_xd9oes
2022-11-23T02:05:07.7522834Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0_xd9oes/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7523043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7523299Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6d6juz7t
2022-11-23T02:05:07.7523569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6d6juz7t/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7523800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7523901Z ok (4.335s)
2022-11-23T02:05:07.7523964Z 
2022-11-23T02:05:07.7524240Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7524351Z Ran 1 test in 4.335s
2022-11-23T02:05:07.7524371Z 
2022-11-23T02:05:07.7524463Z OK
2022-11-23T02:05:07.7524481Z 
2022-11-23T02:05:07.7524586Z Generating XML reports...
2022-11-23T02:05:07.7525026Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020113.xml
2022-11-23T02:05:07.7525399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7525575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7525956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7526150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7526410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzt6t93tj
2022-11-23T02:05:07.7526681Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzt6t93tj/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7526701Z 
2022-11-23T02:05:07.7526807Z Running tests...
2022-11-23T02:05:07.7527055Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7527372Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7527619Z   test_broadcast_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7527839Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23487
2022-11-23T02:05:07.7528060Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23488
2022-11-23T02:05:07.7528283Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23489
2022-11-23T02:05:07.7528499Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23490
2022-11-23T02:05:07.7528878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7529035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7529416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7529609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7529973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7530145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7530516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7530796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7531182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7531338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7531709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7531898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7532262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7532436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7532807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7532997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7533258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptnf3wyfl
2022-11-23T02:05:07.7533592Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptnf3wyfl/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7533804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7534060Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9pt0trbp
2022-11-23T02:05:07.7534330Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9pt0trbp/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7534584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpefo8tcin
2022-11-23T02:05:07.7534855Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpefo8tcin/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7535085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7535312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7535569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqsvvycg7
2022-11-23T02:05:07.7535837Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqsvvycg7/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7536044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7536143Z ok (4.210s)
2022-11-23T02:05:07.7536163Z 
2022-11-23T02:05:07.7536433Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7536544Z Ran 1 test in 4.211s
2022-11-23T02:05:07.7536564Z 
2022-11-23T02:05:07.7536657Z OK
2022-11-23T02:05:07.7536676Z 
2022-11-23T02:05:07.7537268Z Generating XML reports...
2022-11-23T02:05:07.7537741Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020120.xml
2022-11-23T02:05:07.7538119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7538282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7538666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7538856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7539113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprydh693_
2022-11-23T02:05:07.7539381Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprydh693_/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7539402Z 
2022-11-23T02:05:07.7539510Z Running tests...
2022-11-23T02:05:07.7539773Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7540178Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7540546Z   test_broadcast_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7540760Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23830
2022-11-23T02:05:07.7540981Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23831
2022-11-23T02:05:07.7541198Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23832
2022-11-23T02:05:07.7541410Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23833
2022-11-23T02:05:07.7541804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7541980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7542362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7542558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7542904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7543158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7543543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7543735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7544099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7544274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7544650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7544843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7545188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7545365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7545742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7545929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7546187Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpak97n_ka
2022-11-23T02:05:07.7546457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpak97n_ka/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7546686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7546942Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp42avbarc
2022-11-23T02:05:07.7547218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp42avbarc/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7547458Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ep80_65
2022-11-23T02:05:07.7547723Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ep80_65/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7547950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7548205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf2asv9bp
2022-11-23T02:05:07.7548473Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf2asv9bp/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7548702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7548929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7549030Z ok (6.167s)
2022-11-23T02:05:07.7549050Z 
2022-11-23T02:05:07.7549373Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7549477Z Ran 1 test in 6.167s
2022-11-23T02:05:07.7549497Z 
2022-11-23T02:05:07.7549588Z OK
2022-11-23T02:05:07.7549607Z 
2022-11-23T02:05:07.7549732Z Generating XML reports...
2022-11-23T02:05:07.7550172Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020127.xml
2022-11-23T02:05:07.7550541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7550717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7551100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7551294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7551536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwtdmjy76
2022-11-23T02:05:07.7551809Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwtdmjy76/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7551879Z 
2022-11-23T02:05:07.7551990Z Running tests...
2022-11-23T02:05:07.7552327Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7552640Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7552892Z   test_broadcast_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7553115Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24177
2022-11-23T02:05:07.7553332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24178
2022-11-23T02:05:07.7553531Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24179
2022-11-23T02:05:07.7553747Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24180
2022-11-23T02:05:07.7554124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7554301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7554680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7554870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7555231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7555405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7555776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7555946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7556309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7556483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7556860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7557047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7557407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7557580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7557950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7558122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7558429Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeqwcz8qc
2022-11-23T02:05:07.7558705Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeqwcz8qc/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7558937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7559195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcasqfkrf
2022-11-23T02:05:07.7559466Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcasqfkrf/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7559721Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpov_noe1m
2022-11-23T02:05:07.7559987Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpov_noe1m/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7560213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7560422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7560678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4sqb7i94
2022-11-23T02:05:07.7560994Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4sqb7i94/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7561223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7561325Z ok (4.366s)
2022-11-23T02:05:07.7561344Z 
2022-11-23T02:05:07.7561613Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7561726Z Ran 1 test in 4.366s
2022-11-23T02:05:07.7561745Z 
2022-11-23T02:05:07.7561836Z OK
2022-11-23T02:05:07.7561855Z 
2022-11-23T02:05:07.7561963Z Generating XML reports...
2022-11-23T02:05:07.7562406Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020136.xml
2022-11-23T02:05:07.7562780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7562956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7563336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7563525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7563779Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfr4nif7f
2022-11-23T02:05:07.7564050Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfr4nif7f/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7564070Z 
2022-11-23T02:05:07.7564178Z Running tests...
2022-11-23T02:05:07.7564423Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7564737Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7564989Z   test_broadcast_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7565213Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24520
2022-11-23T02:05:07.7565435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24521
2022-11-23T02:05:07.7565650Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24522
2022-11-23T02:05:07.7565864Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24523
2022-11-23T02:05:07.7566238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7566395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7566777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7566969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7567383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7567561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7567945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7568135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7568495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7568668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7569023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7569212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7569573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7569747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7570195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7570381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7570638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqiwdcwhj
2022-11-23T02:05:07.7570912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqiwdcwhj/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7571150Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5sd854h3
2022-11-23T02:05:07.7571424Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5sd854h3/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7571655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7571887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7572140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp73_z9p04
2022-11-23T02:05:07.7572409Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp73_z9p04/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7572635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7572894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgrjepc2g
2022-11-23T02:05:07.7573165Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgrjepc2g/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7573374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7573475Z ok (4.471s)
2022-11-23T02:05:07.7573497Z 
2022-11-23T02:05:07.7573766Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7573879Z Ran 1 test in 4.471s
2022-11-23T02:05:07.7573902Z 
2022-11-23T02:05:07.7573994Z OK
2022-11-23T02:05:07.7574012Z 
2022-11-23T02:05:07.7574139Z Generating XML reports...
2022-11-23T02:05:07.7574578Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020143.xml
2022-11-23T02:05:07.7574948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7575105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7575488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7575677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7575933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpms41h92w
2022-11-23T02:05:07.7576272Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpms41h92w/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7576294Z 
2022-11-23T02:05:07.7576407Z Running tests...
2022-11-23T02:05:07.7576675Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7577272Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7577531Z   test_broadcast_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7577734Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24887
2022-11-23T02:05:07.7577956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24888
2022-11-23T02:05:07.7578173Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24889
2022-11-23T02:05:07.7578389Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24890
2022-11-23T02:05:07.7578770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7578947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7579413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7579606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7579953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7580127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7580497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7580688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7581056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7581235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7581610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7581805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7582172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7582328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7582700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7582888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7583148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpils6h9pw
2022-11-23T02:05:07.7583423Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpils6h9pw/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7583655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7583915Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyjufj99e
2022-11-23T02:05:07.7584186Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyjufj99e/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7584396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7584653Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc7b1qfmc
2022-11-23T02:05:07.7584920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc7b1qfmc/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7585148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7585405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1pd1jku9
2022-11-23T02:05:07.7585742Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1pd1jku9/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7585982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7586084Z ok (6.255s)
2022-11-23T02:05:07.7586105Z 
2022-11-23T02:05:07.7586377Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7586471Z Ran 1 test in 6.255s
2022-11-23T02:05:07.7586490Z 
2022-11-23T02:05:07.7586582Z OK
2022-11-23T02:05:07.7586601Z 
2022-11-23T02:05:07.7586725Z Generating XML reports...
2022-11-23T02:05:07.7587162Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020149.xml
2022-11-23T02:05:07.7587533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7587708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7588091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7588340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7588579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmq7t6qoa
2022-11-23T02:05:07.7588853Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmq7t6qoa/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7588876Z 
2022-11-23T02:05:07.7588988Z Running tests...
2022-11-23T02:05:07.7589255Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7589573Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7589816Z   test_empty_tensors (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7590035Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25258
2022-11-23T02:05:07.7590259Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25259
2022-11-23T02:05:07.7590477Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25260
2022-11-23T02:05:07.7590676Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25261
2022-11-23T02:05:07.7591051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7591225Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7591606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7591804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7592173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7592350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7592732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7592907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7593273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7593445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7593816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7594005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7594374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7594549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7594970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7595172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7595413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe483n0d3
2022-11-23T02:05:07.7595688Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe483n0d3/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7595946Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp56efadk4
2022-11-23T02:05:07.7596216Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp56efadk4/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7596451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7596685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7596943Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5bsqf8xd
2022-11-23T02:05:07.7597222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5bsqf8xd/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7597483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7597743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuehsmupb
2022-11-23T02:05:07.7598023Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuehsmupb/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7598256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7598363Z ok (4.348s)
2022-11-23T02:05:07.7598383Z 
2022-11-23T02:05:07.7598658Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7598774Z Ran 1 test in 4.348s
2022-11-23T02:05:07.7598794Z 
2022-11-23T02:05:07.7598891Z OK
2022-11-23T02:05:07.7598910Z 
2022-11-23T02:05:07.7599039Z Generating XML reports...
2022-11-23T02:05:07.7599460Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020158.xml
2022-11-23T02:05:07.7599839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7600018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7600401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7600594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7600854Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpprn2ho05
2022-11-23T02:05:07.7601130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpprn2ho05/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7601150Z 
2022-11-23T02:05:07.7601263Z Running tests...
2022-11-23T02:05:07.7601513Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7601827Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7602078Z   test_gather_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7602303Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25601
2022-11-23T02:05:07.7602527Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25602
2022-11-23T02:05:07.7602744Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25603
2022-11-23T02:05:07.7602965Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25604
2022-11-23T02:05:07.7603343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7603500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7603941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7604147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7604523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7604702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7605084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7605278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7605643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7605819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7606176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7606366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7606790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7606972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7607351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7607542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7607807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppfl53rcj
2022-11-23T02:05:07.7608088Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppfl53rcj/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7608323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7608564Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx9d5nybi
2022-11-23T02:05:07.7608843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx9d5nybi/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7609080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7609337Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptohad53r
2022-11-23T02:05:07.7609607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptohad53r/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7609838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7610095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2rmsizye
2022-11-23T02:05:07.7610367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2rmsizye/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7610578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7610683Z ok (4.269s)
2022-11-23T02:05:07.7610709Z 
2022-11-23T02:05:07.7610986Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7611105Z Ran 1 test in 4.270s
2022-11-23T02:05:07.7611124Z 
2022-11-23T02:05:07.7611220Z OK
2022-11-23T02:05:07.7611239Z 
2022-11-23T02:05:07.7611363Z Generating XML reports...
2022-11-23T02:05:07.7611799Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020205.xml
2022-11-23T02:05:07.7612173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7612349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7612713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7612959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7613227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl_xhe9on
2022-11-23T02:05:07.7613504Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl_xhe9on/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7613524Z 
2022-11-23T02:05:07.7613642Z Running tests...
2022-11-23T02:05:07.7613910Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7614225Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7614478Z   test_gather_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7614681Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25944
2022-11-23T02:05:07.7614902Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25945
2022-11-23T02:05:07.7615128Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25946
2022-11-23T02:05:07.7615344Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25947
2022-11-23T02:05:07.7615780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7615962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7616353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7616543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7617135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7617316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7617702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7617903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7618273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7618453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7618827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7619019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7619383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7619539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7619906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7620094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7620351Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5wxita4h
2022-11-23T02:05:07.7620626Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5wxita4h/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7620855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7621117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw3t9c9a_
2022-11-23T02:05:07.7621386Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw3t9c9a_/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7621623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpox5zfkvw
2022-11-23T02:05:07.7621892Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpox5zfkvw/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7622193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7622437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7622695Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxesmk8d7
2022-11-23T02:05:07.7622966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxesmk8d7/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7623196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7623298Z ok (6.149s)
2022-11-23T02:05:07.7623319Z 
2022-11-23T02:05:07.7623589Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7623682Z Ran 1 test in 6.149s
2022-11-23T02:05:07.7623702Z 
2022-11-23T02:05:07.7623793Z OK
2022-11-23T02:05:07.7623812Z 
2022-11-23T02:05:07.7623935Z Generating XML reports...
2022-11-23T02:05:07.7624372Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020212.xml
2022-11-23T02:05:07.7624745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7624987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7625370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7625560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7625796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphee0dymc
2022-11-23T02:05:07.7626071Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphee0dymc/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7626090Z 
2022-11-23T02:05:07.7626198Z Running tests...
2022-11-23T02:05:07.7626463Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7626779Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7627020Z   test_gather_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7627243Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26291
2022-11-23T02:05:07.7627458Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26292
2022-11-23T02:05:07.7627675Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26293
2022-11-23T02:05:07.7627871Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26294
2022-11-23T02:05:07.7628250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7628426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7628809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7629006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7629371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7629548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7629923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7630097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7630464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7630636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7631004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7631188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7631654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7631841Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7632213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7632400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7632639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq9nyscyw
2022-11-23T02:05:07.7632894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4q8x4y8g
2022-11-23T02:05:07.7633167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq9nyscyw/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7633435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4q8x4y8g/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7633671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7633944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7634198Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuqqjaeum
2022-11-23T02:05:07.7634471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuqqjaeum/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7634680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7634936Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfjnprfy0
2022-11-23T02:05:07.7635206Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfjnprfy0/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7635436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7635538Z ok (4.196s)
2022-11-23T02:05:07.7635557Z 
2022-11-23T02:05:07.7635831Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7635946Z Ran 1 test in 4.196s
2022-11-23T02:05:07.7635965Z 
2022-11-23T02:05:07.7636186Z OK
2022-11-23T02:05:07.7636205Z 
2022-11-23T02:05:07.7636328Z Generating XML reports...
2022-11-23T02:05:07.7636748Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020221.xml
2022-11-23T02:05:07.7637119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7637295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7637680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7637874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7638136Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5f9vd7kq
2022-11-23T02:05:07.7638412Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5f9vd7kq/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7638435Z 
2022-11-23T02:05:07.7638549Z Running tests...
2022-11-23T02:05:07.7638795Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7639115Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7639381Z   test_gather_noncontiguous_input (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7639607Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26634
2022-11-23T02:05:07.7639829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26635
2022-11-23T02:05:07.7640050Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26636
2022-11-23T02:05:07.7640269Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26637
2022-11-23T02:05:07.7640817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7641002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7641372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7641561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7641929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7642104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7642482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7642672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7643038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7643263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7643622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7643808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7644178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7644355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7644730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7644926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7645190Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_rwkn1i5
2022-11-23T02:05:07.7645466Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_rwkn1i5/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7645728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8g88nxcm
2022-11-23T02:05:07.7645981Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8g88nxcm/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7646216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7646446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7646703Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprw4of0p5
2022-11-23T02:05:07.7646977Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprw4of0p5/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7647206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7647470Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6arju5p8
2022-11-23T02:05:07.7647746Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6arju5p8/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7647956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7648067Z ok (4.350s)
2022-11-23T02:05:07.7648086Z 
2022-11-23T02:05:07.7648366Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7648480Z Ran 1 test in 4.350s
2022-11-23T02:05:07.7648500Z 
2022-11-23T02:05:07.7648595Z OK
2022-11-23T02:05:07.7648614Z 
2022-11-23T02:05:07.7648749Z Generating XML reports...
2022-11-23T02:05:07.7649192Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020227.xml
2022-11-23T02:05:07.7649622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7649811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7650178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7650372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7650634Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpph9wsdth
2022-11-23T02:05:07.7650911Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpph9wsdth/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7650930Z 
2022-11-23T02:05:07.7651044Z Running tests...
2022-11-23T02:05:07.7651308Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7651620Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7651864Z   test_gather_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7652069Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26977
2022-11-23T02:05:07.7652346Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26978
2022-11-23T02:05:07.7652560Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26979
2022-11-23T02:05:07.7652779Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26980
2022-11-23T02:05:07.7653160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7653337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7653717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7653910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7654279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7654435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7654812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7655001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7655363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7655534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7655911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7656097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7656451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7684026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7684552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7684753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7685020Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp915k493c
2022-11-23T02:05:07.7685281Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp915k493c/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7685518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7685784Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn2yhu89y
2022-11-23T02:05:07.7686064Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn2yhu89y/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7686483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_jmwbr77
2022-11-23T02:05:07.7686778Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_jmwbr77/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7687022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7687257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7687501Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp12tiiow1
2022-11-23T02:05:07.7687773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp12tiiow1/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7688011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7688112Z ok (4.847s)
2022-11-23T02:05:07.7688134Z 
2022-11-23T02:05:07.7688429Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7688545Z Ran 1 test in 4.848s
2022-11-23T02:05:07.7688569Z 
2022-11-23T02:05:07.7688661Z OK
2022-11-23T02:05:07.7688681Z 
2022-11-23T02:05:07.7688893Z Generating XML reports...
2022-11-23T02:05:07.7689344Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020234.xml
2022-11-23T02:05:07.7689742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7689925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7690326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7690525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7690788Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj3bl2lsp
2022-11-23T02:05:07.7691066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj3bl2lsp/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7691087Z 
2022-11-23T02:05:07.7691186Z Running tests...
2022-11-23T02:05:07.7691463Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7691776Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7692026Z   test_gather_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7692244Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27344
2022-11-23T02:05:07.7692473Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27345
2022-11-23T02:05:07.7692697Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27346
2022-11-23T02:05:07.7692923Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27347
2022-11-23T02:05:07.7693319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7693506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7693902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7694100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7694495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7694676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7695078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7695272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7695659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7695895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7696310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7696493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7697203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7697401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7697814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7698009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7698278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk3nuq5up
2022-11-23T02:05:07.7698563Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk3nuq5up/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7698826Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxgl9jv1r
2022-11-23T02:05:07.7699247Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxgl9jv1r/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7699471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7699713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7699976Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf0dnoa5q
2022-11-23T02:05:07.7700250Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf0dnoa5q/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7700489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7700746Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3eymq6l0
2022-11-23T02:05:07.7701021Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3eymq6l0/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7701259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7701343Z ok (7.678s)
2022-11-23T02:05:07.7701363Z 
2022-11-23T02:05:07.7701653Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7701761Z Ran 1 test in 7.678s
2022-11-23T02:05:07.7701781Z 
2022-11-23T02:05:07.7701873Z OK
2022-11-23T02:05:07.7701893Z 
2022-11-23T02:05:07.7702015Z Generating XML reports...
2022-11-23T02:05:07.7702470Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020241.xml
2022-11-23T02:05:07.7702855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7703029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7703426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7703610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7703866Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsbnrnmv4
2022-11-23T02:05:07.7704137Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsbnrnmv4/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7704158Z 
2022-11-23T02:05:07.7704266Z Running tests...
2022-11-23T02:05:07.7704548Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7704873Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7705142Z   test_multi_device_constructor (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7705362Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27715
2022-11-23T02:05:07.7705643Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27716
2022-11-23T02:05:07.7705877Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27717
2022-11-23T02:05:07.7706094Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27718
2022-11-23T02:05:07.7706480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7706650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7707043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7707230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7707609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7707774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7708171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7708417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7708793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7708963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7709352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7709538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7709915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7710086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7710475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7710663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7710918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsl5l1f80
2022-11-23T02:05:07.7711186Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsl5l1f80/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7711437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuol3q7a4
2022-11-23T02:05:07.7711703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuol3q7a4/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7711930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7712158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7712411Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjxzu7rpi
2022-11-23T02:05:07.7712671Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjxzu7rpi/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7712925Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpum0rto_h
2022-11-23T02:05:07.7713191Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpum0rto_h/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7713418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7713646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7713736Z ok (4.348s)
2022-11-23T02:05:07.7713756Z 
2022-11-23T02:05:07.7714033Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7714136Z Ran 1 test in 4.349s
2022-11-23T02:05:07.7714156Z 
2022-11-23T02:05:07.7714229Z OK
2022-11-23T02:05:07.7714248Z 
2022-11-23T02:05:07.7714412Z Generating XML reports...
2022-11-23T02:05:07.7714868Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020252.xml
2022-11-23T02:05:07.7715252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7715421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7715810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7715996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7716249Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmilxpinf
2022-11-23T02:05:07.7716512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmilxpinf/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7716538Z 
2022-11-23T02:05:07.7716631Z Running tests...
2022-11-23T02:05:07.7716907Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7717224Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7717522Z   test_reduce_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7717742Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28062
2022-11-23T02:05:07.7717959Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28063
2022-11-23T02:05:07.7718177Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28064
2022-11-23T02:05:07.7718390Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28065
2022-11-23T02:05:07.7718772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7718943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7719348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7719547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7719940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7720120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7720522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7720719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7721092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7721274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7721681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7721880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7722273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7722454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7722853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7723048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7723313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi7k1_mrl
2022-11-23T02:05:07.7723571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi7k1_mrl/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7723833Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeol_mp3q
2022-11-23T02:05:07.7724162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeol_mp3q/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7724414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7724653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7724918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3sczpl2h
2022-11-23T02:05:07.7725195Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3sczpl2h/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7725457Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg0o7ruf4
2022-11-23T02:05:07.7725715Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg0o7ruf4/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7725953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7726194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7726353Z ok (4.438s)
2022-11-23T02:05:07.7726373Z 
2022-11-23T02:05:07.7726661Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7726776Z Ran 1 test in 4.438s
2022-11-23T02:05:07.7726797Z 
2022-11-23T02:05:07.7726889Z OK
2022-11-23T02:05:07.7726909Z 
2022-11-23T02:05:07.7727039Z Generating XML reports...
2022-11-23T02:05:07.7727495Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020258.xml
2022-11-23T02:05:07.7727868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7728050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7728452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7728654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7728920Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpun9igxlq
2022-11-23T02:05:07.7729207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpun9igxlq/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7729227Z 
2022-11-23T02:05:07.7729337Z Running tests...
2022-11-23T02:05:07.7729622Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7729937Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7730199Z   test_reduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7730430Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28405
2022-11-23T02:05:07.7730661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28406
2022-11-23T02:05:07.7730891Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28407
2022-11-23T02:05:07.7731117Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28408
2022-11-23T02:05:07.7731518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7731699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7732103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7732284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7732674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7732855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7733327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7733536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7733936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7734122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7734523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7734704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7735094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7735277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7735677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7735877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7736142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu9t4lzx7
2022-11-23T02:05:07.7736477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu9t4lzx7/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7737019Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiey97xc5
2022-11-23T02:05:07.7737313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiey97xc5/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7737560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyyxfgryz
2022-11-23T02:05:07.7737842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyyxfgryz/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7738080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7738324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7738564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7738831Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg9jftib4
2022-11-23T02:05:07.7739107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg9jftib4/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7739346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7739430Z ok (6.136s)
2022-11-23T02:05:07.7739466Z 
2022-11-23T02:05:07.7739748Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7739865Z Ran 1 test in 6.136s
2022-11-23T02:05:07.7739885Z 
2022-11-23T02:05:07.7739977Z OK
2022-11-23T02:05:07.7739997Z 
2022-11-23T02:05:07.7740122Z Generating XML reports...
2022-11-23T02:05:07.7740679Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020305.xml
2022-11-23T02:05:07.7741074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7741254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7741634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7741808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7742062Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2g10umr9
2022-11-23T02:05:07.7742332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2g10umr9/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7742352Z 
2022-11-23T02:05:07.7742459Z Running tests...
2022-11-23T02:05:07.7742728Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7743122Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7743379Z   test_reduce_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7743602Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28752
2022-11-23T02:05:07.7743803Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28753
2022-11-23T02:05:07.7744017Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28754
2022-11-23T02:05:07.7744231Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28755
2022-11-23T02:05:07.7744610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7744787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7745169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7745363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7745729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7746011Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7746377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7746567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7746930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7747104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7747477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7747664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7748029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7748205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7748559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7748747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7749001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy1q48nqd
2022-11-23T02:05:07.7749270Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy1q48nqd/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7749520Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp25u18j29
2022-11-23T02:05:07.7749783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp25u18j29/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7750036Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplpddhgsa
2022-11-23T02:05:07.7750305Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplpddhgsa/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7750533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7750742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7750966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7751218Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwfgd7jsx
2022-11-23T02:05:07.7751484Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwfgd7jsx/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7751706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7751806Z ok (4.347s)
2022-11-23T02:05:07.7751826Z 
2022-11-23T02:05:07.7752156Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7752281Z Ran 1 test in 4.347s
2022-11-23T02:05:07.7752300Z 
2022-11-23T02:05:07.7752374Z OK
2022-11-23T02:05:07.7752393Z 
2022-11-23T02:05:07.7752517Z Generating XML reports...
2022-11-23T02:05:07.7752958Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020314.xml
2022-11-23T02:05:07.7753327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7753504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7753879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7754071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7754326Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9kjn_6on
2022-11-23T02:05:07.7754591Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9kjn_6on/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7754662Z 
2022-11-23T02:05:07.7754758Z Running tests...
2022-11-23T02:05:07.7755031Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7755340Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7755582Z   test_reduce_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7755802Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29095
2022-11-23T02:05:07.7756018Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29096
2022-11-23T02:05:07.7756234Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29097
2022-11-23T02:05:07.7756447Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29098
2022-11-23T02:05:07.7756804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7756982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7757361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7757549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7757914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7758090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7758465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7758656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7759025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7759186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7759564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7759752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7760114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7760291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7760665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7760852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7761159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3weyvdt4
2022-11-23T02:05:07.7761422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3weyvdt4/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7761656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7761911Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph5w0lddq
2022-11-23T02:05:07.7762187Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph5w0lddq/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7762417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7762668Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2n8ipq44
2022-11-23T02:05:07.7762932Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2n8ipq44/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7763185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppmlgsfrt
2022-11-23T02:05:07.7763464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppmlgsfrt/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7763723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7763954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7764058Z ok (4.550s)
2022-11-23T02:05:07.7764079Z 
2022-11-23T02:05:07.7764348Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7764460Z Ran 1 test in 4.550s
2022-11-23T02:05:07.7764479Z 
2022-11-23T02:05:07.7764573Z OK
2022-11-23T02:05:07.7764591Z 
2022-11-23T02:05:07.7764715Z Generating XML reports...
2022-11-23T02:05:07.7765146Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020321.xml
2022-11-23T02:05:07.7765506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7765680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7766063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7766253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7766507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkalcrd98
2022-11-23T02:05:07.7766776Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkalcrd98/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7766796Z 
2022-11-23T02:05:07.7766903Z Running tests...
2022-11-23T02:05:07.7767165Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7767474Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7767711Z   test_reduce_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7767934Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29462
2022-11-23T02:05:07.7768155Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29463
2022-11-23T02:05:07.7768369Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29464
2022-11-23T02:05:07.7768581Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29465
2022-11-23T02:05:07.7768956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7769130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7769509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7769684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7770100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7770285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7770669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7770859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7771223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7771395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7771770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7771958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7772303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7772480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7772917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7773104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7773360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvo67srgo
2022-11-23T02:05:07.7773629Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvo67srgo/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7773882Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfqcbainu
2022-11-23T02:05:07.7774153Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfqcbainu/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7774388Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz0hb064c
2022-11-23T02:05:07.7774652Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz0hb064c/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7774881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7775106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7775331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7775585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkn8utq_2
2022-11-23T02:05:07.7775849Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkn8utq_2/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7776071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7776222Z ok (6.726s)
2022-11-23T02:05:07.7776260Z 
2022-11-23T02:05:07.7776513Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7776625Z Ran 1 test in 6.726s
2022-11-23T02:05:07.7776649Z 
2022-11-23T02:05:07.7777166Z OK
2022-11-23T02:05:07.7777189Z 
2022-11-23T02:05:07.7777329Z Generating XML reports...
2022-11-23T02:05:07.7777775Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020328.xml
2022-11-23T02:05:07.7778146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7778322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7778698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7778868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7779122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpar80hrrz
2022-11-23T02:05:07.7779489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpar80hrrz/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7779512Z 
2022-11-23T02:05:07.7779633Z Running tests...
2022-11-23T02:05:07.7779906Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7780217Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7780455Z   test_round_robin (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7780672Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29833
2022-11-23T02:05:07.7780870Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29834
2022-11-23T02:05:07.7781084Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29835
2022-11-23T02:05:07.7781298Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29836
2022-11-23T02:05:07.7781670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7781850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7782306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7782500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7782868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7783044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7783404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7783593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7783957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7784134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7784506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7784695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7785054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7785226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7785584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7785771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7786025Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgbsbgm1i
2022-11-23T02:05:07.7786299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgbsbgm1i/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7786554Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqkg55kt8
2022-11-23T02:05:07.7786823Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqkg55kt8/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7787073Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprnu2bekq
2022-11-23T02:05:07.7787337Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprnu2bekq/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7787566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7787776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7788000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7788254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuwlbpam4
2022-11-23T02:05:07.7788571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuwlbpam4/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7788810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7789383Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-11-23T02:05:07.7789929Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-11-23T02:05:07.7790464Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-11-23T02:05:07.7791053Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-11-23T02:05:07.7791156Z ok (4.326s)
2022-11-23T02:05:07.7791176Z 
2022-11-23T02:05:07.7791450Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7791563Z Ran 1 test in 4.326s
2022-11-23T02:05:07.7791583Z 
2022-11-23T02:05:07.7791654Z OK
2022-11-23T02:05:07.7791673Z 
2022-11-23T02:05:07.7791796Z Generating XML reports...
2022-11-23T02:05:07.7792233Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020337.xml
2022-11-23T02:05:07.7792605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7792785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7793162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7793353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7793611Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt9zlb7ks
2022-11-23T02:05:07.7793862Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt9zlb7ks/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7793900Z 
2022-11-23T02:05:07.7793990Z Running tests...
2022-11-23T02:05:07.7794253Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7794566Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7794907Z   test_round_robin_create_destroy (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7795128Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30188
2022-11-23T02:05:07.7795345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30189
2022-11-23T02:05:07.7795559Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30190
2022-11-23T02:05:07.7795772Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30191
2022-11-23T02:05:07.7796132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7796307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7796684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7796941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7797322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7797501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7797875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7798063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7798413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7798585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7798956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7799145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7799508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7799737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7800112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7800300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7800555Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjy_n5s3p
2022-11-23T02:05:07.7800807Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjy_n5s3p/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7801061Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjj6n_6hx
2022-11-23T02:05:07.7801414Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjj6n_6hx/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7801657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7801890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7802140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph_1vc96_
2022-11-23T02:05:07.7802402Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph_1vc96_/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7802653Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ml0giq6
2022-11-23T02:05:07.7802919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ml0giq6/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7803130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7803354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7803911Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-11-23T02:05:07.7804462Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-11-23T02:05:07.7804991Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-11-23T02:05:07.7805582Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-11-23T02:05:07.7806132Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-11-23T02:05:07.7806663Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-11-23T02:05:07.7807193Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-11-23T02:05:07.7807765Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-11-23T02:05:07.7807867Z ok (4.444s)
2022-11-23T02:05:07.7807888Z 
2022-11-23T02:05:07.7808163Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7808274Z Ran 1 test in 4.444s
2022-11-23T02:05:07.7808294Z 
2022-11-23T02:05:07.7808386Z OK
2022-11-23T02:05:07.7808405Z 
2022-11-23T02:05:07.7808530Z Generating XML reports...
2022-11-23T02:05:07.7808951Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020344.xml
2022-11-23T02:05:07.7809322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7809500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7809882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7810074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7810329Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpel8ot6m6
2022-11-23T02:05:07.7810598Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpel8ot6m6/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7810619Z 
2022-11-23T02:05:07.7810726Z Running tests...
2022-11-23T02:05:07.7810993Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7811293Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7811539Z   test_scatter_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7811761Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30567
2022-11-23T02:05:07.7811978Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30568
2022-11-23T02:05:07.7812193Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30569
2022-11-23T02:05:07.7812406Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30570
2022-11-23T02:05:07.7812781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7812954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7813314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7813559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7813942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7814120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7814497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7814685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7815049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7815222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7815580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7815773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7816140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7816371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7816991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7817194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7817453Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppitx52wy
2022-11-23T02:05:07.7817727Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppitx52wy/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7817957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7818197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr57bg9pu
2022-11-23T02:05:07.7818471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr57bg9pu/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7818734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1mmj_z3z
2022-11-23T02:05:07.7818999Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1mmj_z3z/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7819250Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvlsloix8
2022-11-23T02:05:07.7819513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvlsloix8/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7819741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7819968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7820190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7820274Z ok (4.261s)
2022-11-23T02:05:07.7820299Z 
2022-11-23T02:05:07.7820581Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7820697Z Ran 1 test in 4.261s
2022-11-23T02:05:07.7820716Z 
2022-11-23T02:05:07.7820809Z OK
2022-11-23T02:05:07.7820827Z 
2022-11-23T02:05:07.7820951Z Generating XML reports...
2022-11-23T02:05:07.7821386Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020351.xml
2022-11-23T02:05:07.7821757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7821931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7822297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7822488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7822863Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_fceqow2
2022-11-23T02:05:07.7823144Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_fceqow2/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7823167Z 
2022-11-23T02:05:07.7823279Z Running tests...
2022-11-23T02:05:07.7823548Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7823861Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7824118Z   test_scatter_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7824316Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30910
2022-11-23T02:05:07.7824534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30911
2022-11-23T02:05:07.7824746Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30912
2022-11-23T02:05:07.7824964Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30913
2022-11-23T02:05:07.7825341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7825589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7825972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7826162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7826530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7826686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7827059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7827246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7827612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7827790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7828164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7828351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7828715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7828870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7829245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7829433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7829693Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo_wpje1s
2022-11-23T02:05:07.7829964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo_wpje1s/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7830217Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqmxxnll9
2022-11-23T02:05:07.7830485Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqmxxnll9/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7830720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7830952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7831183Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpofba3xu1
2022-11-23T02:05:07.7831446Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpofba3xu1/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7831746Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbj5bfce1
2022-11-23T02:05:07.7832021Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbj5bfce1/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7832259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7832494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7832600Z ok (6.108s)
2022-11-23T02:05:07.7832621Z 
2022-11-23T02:05:07.7832902Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7832997Z Ran 1 test in 6.108s
2022-11-23T02:05:07.7833038Z 
2022-11-23T02:05:07.7833112Z OK
2022-11-23T02:05:07.7833131Z 
2022-11-23T02:05:07.7833262Z Generating XML reports...
2022-11-23T02:05:07.7833701Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020357.xml
2022-11-23T02:05:07.7834082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7834263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7834708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7834907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7835167Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbqzq1gqe
2022-11-23T02:05:07.7835420Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbqzq1gqe/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7835439Z 
2022-11-23T02:05:07.7835551Z Running tests...
2022-11-23T02:05:07.7835825Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7836139Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7836390Z   test_scatter_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7836613Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31257
2022-11-23T02:05:07.7836840Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31258
2022-11-23T02:05:07.7837058Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31259
2022-11-23T02:05:07.7837255Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31260
2022-11-23T02:05:07.7837631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7837810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7838190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7838385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7838762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7838942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7839326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7839518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7839865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7840041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7840511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7840717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7841194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7841382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7841769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7841962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7842202Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4p3jhrmv
2022-11-23T02:05:07.7842478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4p3jhrmv/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7842712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7842979Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcmpj66uk
2022-11-23T02:05:07.7843252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcmpj66uk/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7843510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp93ijd18p
2022-11-23T02:05:07.7843837Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp93ijd18p/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7844094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppgpe799r
2022-11-23T02:05:07.7844368Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppgpe799r/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7844578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7844808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7845041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7845146Z ok (4.263s)
2022-11-23T02:05:07.7845166Z 
2022-11-23T02:05:07.7845443Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7845562Z Ran 1 test in 4.263s
2022-11-23T02:05:07.7845582Z 
2022-11-23T02:05:07.7845682Z OK
2022-11-23T02:05:07.7845704Z 
2022-11-23T02:05:07.7845835Z Generating XML reports...
2022-11-23T02:05:07.7846254Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020406.xml
2022-11-23T02:05:07.7846635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7846814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7847202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7847395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7847649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyhuff88j
2022-11-23T02:05:07.7847925Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyhuff88j/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7847945Z 
2022-11-23T02:05:07.7848057Z Running tests...
2022-11-23T02:05:07.7848333Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7848631Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7848882Z   test_scatter_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7849104Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31600
2022-11-23T02:05:07.7849326Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31601
2022-11-23T02:05:07.7849545Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31602
2022-11-23T02:05:07.7849763Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31603
2022-11-23T02:05:07.7850191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7850379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7850737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7850917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7851298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7851500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7851880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7852079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7852449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7852637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7853020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7853247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7853625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7853801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7854182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7854375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7854638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqs0srn67
2022-11-23T02:05:07.7854917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqs0srn67/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7855151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7855388Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp37kog90y
2022-11-23T02:05:07.7855659Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp37kog90y/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7855891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7856148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6vdpfhtk
2022-11-23T02:05:07.7856422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6vdpfhtk/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7856678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2gustagq
2022-11-23T02:05:07.7857237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2gustagq/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7857474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7857703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7857787Z ok (4.835s)
2022-11-23T02:05:07.7857808Z 
2022-11-23T02:05:07.7858090Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7858208Z Ran 1 test in 4.835s
2022-11-23T02:05:07.7858229Z 
2022-11-23T02:05:07.7858333Z OK
2022-11-23T02:05:07.7858352Z 
2022-11-23T02:05:07.7858486Z Generating XML reports...
2022-11-23T02:05:07.7858925Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020413.xml
2022-11-23T02:05:07.7859298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7859486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7859936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7860153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7860416Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpulaa9ub3
2022-11-23T02:05:07.7860695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpulaa9ub3/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7860715Z 
2022-11-23T02:05:07.7860826Z Running tests...
2022-11-23T02:05:07.7861099Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7861415Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7861737Z   test_scatter_stress_cuda (__main__.ProcessGroupGlooTest) ... skip: Test is flaky, see https://github.com/pytorch/pytorch/issues/15963 (0.001s)
2022-11-23T02:05:07.7861757Z 
2022-11-23T02:05:07.7862031Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7862125Z Ran 1 test in 0.001s
2022-11-23T02:05:07.7862211Z 
2022-11-23T02:05:07.7862332Z OK (skipped=1)
2022-11-23T02:05:07.7862351Z 
2022-11-23T02:05:07.7862485Z Generating XML reports...
2022-11-23T02:05:07.7862922Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020420.xml
2022-11-23T02:05:07.7863295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7863475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7863861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7864055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7864314Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz8m8e4si
2022-11-23T02:05:07.7864568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz8m8e4si/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7864590Z 
2022-11-23T02:05:07.7864704Z Running tests...
2022-11-23T02:05:07.7864974Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7865290Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7865545Z   test_send_recv_all_to_all (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7865769Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32032
2022-11-23T02:05:07.7865992Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32033
2022-11-23T02:05:07.7866212Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32034
2022-11-23T02:05:07.7866411Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32035
2022-11-23T02:05:07.7866792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7866974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7867359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7867557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7867930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7868106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7868485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7868679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7869079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7869273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7869661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7869853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7870226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7870411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7870793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7870987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7871227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpttxir5x2
2022-11-23T02:05:07.7871506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpttxir5x2/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7871842Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp79r8p065
2022-11-23T02:05:07.7872114Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp79r8p065/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7872372Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpec8leyzg
2022-11-23T02:05:07.7872611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7872888Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpec8leyzg/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7873143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq_hnn74d
2022-11-23T02:05:07.7873409Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq_hnn74d/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7873622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7873859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7874085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7874186Z ok (4.220s)
2022-11-23T02:05:07.7874206Z 
2022-11-23T02:05:07.7874488Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7874609Z Ran 1 test in 4.220s
2022-11-23T02:05:07.7874630Z 
2022-11-23T02:05:07.7874727Z OK
2022-11-23T02:05:07.7874747Z 
2022-11-23T02:05:07.7874874Z Generating XML reports...
2022-11-23T02:05:07.7875291Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020422.xml
2022-11-23T02:05:07.7875666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7875849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7876237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7876437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7876694Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3je5rzc2
2022-11-23T02:05:07.7876966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3je5rzc2/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7876986Z 
2022-11-23T02:05:07.7877101Z Running tests...
2022-11-23T02:05:07.7877372Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7877668Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7877948Z   test_sparse_allreduce_basics (__main__.ProcessGroupGlooTest) ... skip: intermittent failures on Windows, in CI (0.000s)
2022-11-23T02:05:07.7878016Z 
2022-11-23T02:05:07.7878298Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7878421Z Ran 1 test in 0.001s
2022-11-23T02:05:07.7878440Z 
2022-11-23T02:05:07.7878552Z OK (skipped=1)
2022-11-23T02:05:07.7878571Z 
2022-11-23T02:05:07.7878698Z Generating XML reports...
2022-11-23T02:05:07.7879138Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020429.xml
2022-11-23T02:05:07.7879514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7879670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7880051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7880249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7880521Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0qckhuhw
2022-11-23T02:05:07.7880794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0qckhuhw/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7880864Z 
2022-11-23T02:05:07.7880982Z Running tests...
2022-11-23T02:05:07.7881254Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7881568Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7881846Z   test_sparse_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7882050Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32440
2022-11-23T02:05:07.7882271Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32441
2022-11-23T02:05:07.7882489Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32442
2022-11-23T02:05:07.7882709Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32443
2022-11-23T02:05:07.7883091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7883275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7883659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7883853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7884201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7884382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7884765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7884958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7885333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7885514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7885893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7886083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7886449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7886606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7886982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7887173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7887489Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0n2e9dw2
2022-11-23T02:05:07.7887773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0n2e9dw2/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7888013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7888273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkc7ajqau
2022-11-23T02:05:07.7888558Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkc7ajqau/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7888796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgqo5ikuv
2022-11-23T02:05:07.7889068Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgqo5ikuv/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7889302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7889537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7889793Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgu5r010v
2022-11-23T02:05:07.7890114Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgu5r010v/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7890344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7890452Z ok (6.248s)
2022-11-23T02:05:07.7890472Z 
2022-11-23T02:05:07.7890750Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7890844Z Ran 1 test in 6.248s
2022-11-23T02:05:07.7890864Z 
2022-11-23T02:05:07.7890961Z OK
2022-11-23T02:05:07.7890980Z 
2022-11-23T02:05:07.7891113Z Generating XML reports...
2022-11-23T02:05:07.7891549Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020431.xml
2022-11-23T02:05:07.7891928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7892108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7892491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7892688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7892926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptetkqxns
2022-11-23T02:05:07.7893198Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptetkqxns/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7893219Z 
2022-11-23T02:05:07.7893330Z Running tests...
2022-11-23T02:05:07.7893599Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7893919Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7894190Z   test_sparse_allreduce_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7894413Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33159
2022-11-23T02:05:07.7894639Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33160
2022-11-23T02:05:07.7894858Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 33161
2022-11-23T02:05:07.7895053Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 33162
2022-11-23T02:05:07.7895427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7895605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7895985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7896179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7896609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7897062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7897467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7897642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7898013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7898191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7898559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7898734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7899112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7899306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7899781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7899977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7900218Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi7l7acx3
2022-11-23T02:05:07.7900492Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi7l7acx3/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7900748Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4xhw2nfu
2022-11-23T02:05:07.7901026Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4xhw2nfu/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7901262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:05:07.7901497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:05:07.7901759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptktv4h0s
2022-11-23T02:05:07.7902028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptktv4h0s/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7902260Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxl8v6vem
2022-11-23T02:05:07.7902533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxl8v6vem/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7902763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:05:07.7902993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:05:07.7903101Z ok (4.205s)
2022-11-23T02:05:07.7903121Z 
2022-11-23T02:05:07.7903397Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7903515Z Ran 1 test in 4.205s
2022-11-23T02:05:07.7903535Z 
2022-11-23T02:05:07.7903634Z OK
2022-11-23T02:05:07.7903654Z 
2022-11-23T02:05:07.7903782Z Generating XML reports...
2022-11-23T02:05:07.7904197Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020440.xml
2022-11-23T02:05:07.7904569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7904750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7905133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7905328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7905586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3p48joz1
2022-11-23T02:05:07.7905925Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3p48joz1/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7905947Z 
2022-11-23T02:05:07.7906069Z Running tests...
2022-11-23T02:05:07.7906320Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7906639Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7906819Z   test_forward_backward (__main__.ReducerTest) ... ok (0.012s)
2022-11-23T02:05:07.7906838Z 
2022-11-23T02:05:07.7907106Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7907221Z Ran 1 test in 0.022s
2022-11-23T02:05:07.7907241Z 
2022-11-23T02:05:07.7907335Z OK
2022-11-23T02:05:07.7907354Z 
2022-11-23T02:05:07.7907482Z Generating XML reports...
2022-11-23T02:05:07.7907883Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123020447.xml
2022-11-23T02:05:07.7908261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7908417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7908862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7909061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7909317Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkq3c55o4
2022-11-23T02:05:07.7909593Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkq3c55o4/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7909613Z 
2022-11-23T02:05:07.7909725Z Running tests...
2022-11-23T02:05:07.7909993Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7910306Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7911212Z   test_forward_backward_optimizer (__main__.ReducerTest) ... [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:05:07.7911325Z ok (0.018s)
2022-11-23T02:05:07.7911345Z 
2022-11-23T02:05:07.7911596Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7911716Z Ran 1 test in 0.022s
2022-11-23T02:05:07.7911736Z 
2022-11-23T02:05:07.7911831Z OK
2022-11-23T02:05:07.7911849Z 
2022-11-23T02:05:07.7911975Z Generating XML reports...
2022-11-23T02:05:07.7912380Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123020449.xml
2022-11-23T02:05:07.7912753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7912937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7913320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7913515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7913751Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprsai6hd2
2022-11-23T02:05:07.7914027Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprsai6hd2/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7914046Z 
2022-11-23T02:05:07.7914158Z Running tests...
2022-11-23T02:05:07.7914427Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7914795Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7915015Z   test_forward_backward_unused_parameters (__main__.ReducerTest) ... ok (0.014s)
2022-11-23T02:05:07.7915039Z 
2022-11-23T02:05:07.7915307Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7915428Z Ran 1 test in 0.022s
2022-11-23T02:05:07.7915448Z 
2022-11-23T02:05:07.7915522Z OK
2022-11-23T02:05:07.7915568Z 
2022-11-23T02:05:07.7915671Z Generating XML reports...
2022-11-23T02:05:07.7916067Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123020451.xml
2022-11-23T02:05:07.7916439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7916626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7917007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7917208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7917468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd8_q7kkx
2022-11-23T02:05:07.7917801Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd8_q7kkx/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7917821Z 
2022-11-23T02:05:07.7917912Z Running tests...
2022-11-23T02:05:07.7918182Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7918496Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7918685Z   test_multi_dtype_multi_bucket (__main__.ReducerTest) ... ok (0.007s)
2022-11-23T02:05:07.7918704Z 
2022-11-23T02:05:07.7918972Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7919094Z Ran 1 test in 0.012s
2022-11-23T02:05:07.7919114Z 
2022-11-23T02:05:07.7919209Z OK
2022-11-23T02:05:07.7919228Z 
2022-11-23T02:05:07.7919359Z Generating XML reports...
2022-11-23T02:05:07.7919733Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123020453.xml
2022-11-23T02:05:07.7920116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7920296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7920679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7920875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7921137Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxbjvbub4
2022-11-23T02:05:07.7921413Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxbjvbub4/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7921433Z 
2022-11-23T02:05:07.7921543Z Running tests...
2022-11-23T02:05:07.7921818Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7922112Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7922308Z   test_multi_dtype_single_bucket (__main__.ReducerTest) ... ok (0.009s)
2022-11-23T02:05:07.7922327Z 
2022-11-23T02:05:07.7922593Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7922708Z Ran 1 test in 0.012s
2022-11-23T02:05:07.7922727Z 
2022-11-23T02:05:07.7922827Z OK
2022-11-23T02:05:07.7922846Z 
2022-11-23T02:05:07.7922975Z Generating XML reports...
2022-11-23T02:05:07.7923376Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123020456.xml
2022-11-23T02:05:07.7923750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7923930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7924361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7924574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7924837Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptl_uwy2s
2022-11-23T02:05:07.7925113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptl_uwy2s/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7925133Z 
2022-11-23T02:05:07.7925247Z Running tests...
2022-11-23T02:05:07.7925516Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7925833Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7926029Z   test_single_dtype_single_bucket (__main__.ReducerTest) ... ok (0.006s)
2022-11-23T02:05:07.7926048Z 
2022-11-23T02:05:07.7926288Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7926402Z Ran 1 test in 0.012s
2022-11-23T02:05:07.7926426Z 
2022-11-23T02:05:07.7926524Z OK
2022-11-23T02:05:07.7926542Z 
2022-11-23T02:05:07.7926722Z Generating XML reports...
2022-11-23T02:05:07.7927122Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123020458.xml
2022-11-23T02:05:07.7927493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7927673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7928055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7928252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7928488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjllxtp9x
2022-11-23T02:05:07.7928772Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjllxtp9x/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7928792Z 
2022-11-23T02:05:07.7928903Z Running tests...
2022-11-23T02:05:07.7929178Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7929492Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7929724Z   test_logging_init (__main__.RendezvousEnvTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7929974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:05:07.7930385Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:05:07.7930469Z ok (1.790s)
2022-11-23T02:05:07.7930510Z 
2022-11-23T02:05:07.7930756Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7930874Z Ran 1 test in 1.791s
2022-11-23T02:05:07.7930894Z 
2022-11-23T02:05:07.7930989Z OK
2022-11-23T02:05:07.7931008Z 
2022-11-23T02:05:07.7931140Z Generating XML reports...
2022-11-23T02:05:07.7931559Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-RendezvousEnvTest-20221123020500.xml
2022-11-23T02:05:07.7931937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:05:07.7932118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:05:07.7932500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:05:07.7932672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:05:07.7932932Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwtxqgxnz
2022-11-23T02:05:07.7933209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwtxqgxnz/_remote_module_non_scriptable.py
2022-11-23T02:05:07.7933230Z 
2022-11-23T02:05:07.7933342Z Running tests...
2022-11-23T02:05:07.7933658Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7933990Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-11-23T02:05:07.7934237Z   test_default_store_timeout_gloo (__main__.TimeoutTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:05:07.7934991Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/74714 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.736s)
2022-11-23T02:05:07.7935012Z 
2022-11-23T02:05:07.7935278Z ----------------------------------------------------------------------
2022-11-23T02:05:07.7935372Z Ran 1 test in 1.737s
2022-11-23T02:05:07.7935413Z 
2022-11-23T02:05:07.7935502Z OK (skipped=1)
2022-11-23T02:05:07.7935522Z 
2022-11-23T02:05:07.7935656Z Generating XML reports...
2022-11-23T02:05:07.7936060Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-TimeoutTest-20221123020505.xml
2022-11-23T02:05:07.7936128Z 
2022-11-23T02:05:07.7936949Z ##[endgroup]
2022-11-23T02:05:07.7937427Z FINISHED PRINTING LOG FILE of distributed/test_c10d_gloo (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_gloo_yjo3apew)
2022-11-23T02:05:07.7937447Z 
2022-11-23T02:05:08.0241503Z 
2022-11-23T02:05:08.0242114Z real	14m41.984s
2022-11-23T02:05:08.0242407Z user	29m57.421s
2022-11-23T02:05:08.0242670Z sys	23m22.482s
2022-11-23T02:05:08.0243236Z + python test/run_test.py --verbose -i distributed/test_c10d_nccl
2022-11-23T02:05:10.3618281Z Ignoring disabled issues:  []
2022-11-23T02:05:10.4137604Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T02:05:10.4138705Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T02:05:10.4139095Z Selected tests:
2022-11-23T02:05:10.4139368Z  distributed/test_c10d_nccl
2022-11-23T02:05:10.4167022Z Prioritized test from test file changes.
2022-11-23T02:05:10.4167721Z reordering tests for PR:
2022-11-23T02:05:10.4168325Z prioritized: []
2022-11-23T02:05:10.4169004Z the rest: ['distributed/test_c10d_nccl']
2022-11-23T02:05:10.4169208Z 
2022-11-23T02:05:10.4169759Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T02:05:10.4170721Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T02:05:10.4175881Z parallel (file granularity) tests:
2022-11-23T02:05:10.4176338Z  
2022-11-23T02:05:10.4176604Z serial (file granularity) tests:
2022-11-23T02:05:10.4177113Z  distributed/test_c10d_nccl
2022-11-23T02:05:12.7321048Z Ignoring disabled issues:  []
2022-11-23T02:05:12.7516801Z Ignoring disabled issues:  []
2022-11-23T02:05:13.1543157Z Running distributed/test_c10d_nccl ... [2022-11-23 02:05:13.153795]
2022-11-23T02:05:13.1546940Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:05:13.154265]
2022-11-23T02:25:23.0084957Z 
2022-11-23T02:25:23.0085557Z Expand the folded group to see the log file of distributed/test_c10d_nccl
2022-11-23T02:25:23.0087099Z ##[group]PRINTING LOG FILE of distributed/test_c10d_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_nccl_hnzo7h72)
2022-11-23T02:25:23.0089446Z <unittest.suite.TestSuite tests=[<__main__.CommTest testMethod=test_all_reduce_coalesced_nccl>, <__main__.CommTest testMethod=test_broadcast_coalesced_nccl>, <__main__.CommTest testMethod=test_nccl_barrier>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids_function_argument>, <__main__.CommTest testMethod=test_nccl_barrier_timeout>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group_non_member>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_detail>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_info>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_off>, <__main__.CommTest testMethod=test_nncl_rank_membership>, <__main__.CommTest testMethod=test_pass_nccl_options_high_priority_stream>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_default>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_subgroup>, <__main__.CommTest testMethod=test_sequence_num_set_default_pg_nccl>, <__main__.CommTest testMethod=test_sequence_num_set_nccl_new_group>, <__main__.CommTest testMethod=test_tensor_dtype_complex>, <__main__.CommTest testMethod=test_tensor_dtype_mismatch>]>
2022-11-23T02:25:23.0091466Z test_all_reduce_coalesced_nccl (__main__.CommTest)
2022-11-23T02:25:23.0091818Z test_broadcast_coalesced_nccl (__main__.CommTest)
2022-11-23T02:25:23.0092287Z test_nccl_barrier (__main__.CommTest)
2022-11-23T02:25:23.0092617Z test_nccl_barrier_device_ids (__main__.CommTest)
2022-11-23T02:25:23.0092962Z test_nccl_barrier_device_ids_function_argument (__main__.CommTest)
2022-11-23T02:25:23.0093323Z test_nccl_barrier_timeout (__main__.CommTest)
2022-11-23T02:25:23.0093668Z test_nccl_barrier_timeout_new_group (__main__.CommTest)
2022-11-23T02:25:23.0094050Z test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest)
2022-11-23T02:25:23.0094410Z test_nccl_warn_not_in_group_debug_detail (__main__.CommTest)
2022-11-23T02:25:23.0094782Z test_nccl_warn_not_in_group_debug_info (__main__.CommTest)
2022-11-23T02:25:23.0095134Z test_nccl_warn_not_in_group_debug_off (__main__.CommTest)
2022-11-23T02:25:23.0095483Z test_nncl_rank_membership (__main__.CommTest)
2022-11-23T02:25:23.0099972Z test_pass_nccl_options_high_priority_stream (__main__.CommTest)
2022-11-23T02:25:23.0100378Z test_sequence_num_incremented_nccl_default (__main__.CommTest)
2022-11-23T02:25:23.0100793Z test_sequence_num_incremented_nccl_subgroup (__main__.CommTest)
2022-11-23T02:25:23.0101163Z test_sequence_num_set_default_pg_nccl (__main__.CommTest)
2022-11-23T02:25:23.0101509Z test_sequence_num_set_nccl_new_group (__main__.CommTest)
2022-11-23T02:25:23.0101846Z test_tensor_dtype_complex (__main__.CommTest)
2022-11-23T02:25:23.0102176Z test_tensor_dtype_mismatch (__main__.CommTest)
2022-11-23T02:25:23.0103104Z <unittest.suite.TestSuite tests=[<__main__.CompilerTest testMethod=test_allgather_work_wait_gpu>, <__main__.CompilerTest testMethod=test_allreduce_work_wait_gpu>, <__main__.CompilerTest testMethod=test_broadcast_work_wait_gpu>, <__main__.CompilerTest testMethod=test_consecutive_comm_work_wait_gpu>, <__main__.CompilerTest testMethod=test_nested_comm_tensor_wrapping>, <__main__.CompilerTest testMethod=test_reduce_scatter_work_wait_gpu>, <__main__.CompilerTest testMethod=test_scatter_work_wait_gpu>]>
2022-11-23T02:25:23.0104067Z test_allgather_work_wait_gpu (__main__.CompilerTest)
2022-11-23T02:25:23.0104526Z test_allreduce_work_wait_gpu (__main__.CompilerTest)
2022-11-23T02:25:23.0105128Z test_broadcast_work_wait_gpu (__main__.CompilerTest)
2022-11-23T02:25:23.0105579Z test_consecutive_comm_work_wait_gpu (__main__.CompilerTest)
2022-11-23T02:25:23.0105996Z test_nested_comm_tensor_wrapping (__main__.CompilerTest)
2022-11-23T02:25:23.0106704Z test_reduce_scatter_work_wait_gpu (__main__.CompilerTest)
2022-11-23T02:25:23.0107344Z test_scatter_work_wait_gpu (__main__.CompilerTest)
2022-11-23T02:25:23.0122999Z <unittest.suite.TestSuite tests=[<__main__.DistributedDataParallelTest testMethod=test_accumulate_gradients_module>, <__main__.DistributedDataParallelTest testMethod=test_accumulate_gradients_module_with_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_channels_last_contig>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_module>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_static_graph>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_with_then_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_gpu_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_multi_device_module_config>, <__main__.DistributedDataParallelTest testMethod=test_ddp_packed_sequence>, <__main__.DistributedDataParallelTest testMethod=test_ddp_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_with_lazy_parameters>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl_is_view>, <__main__.DistributedDataParallelTest testMethod=test_failure_recovery>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_fp16>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_fp16_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_1devicemodule_1replicaperprocess>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_2devicemodule>, <__main__.DistributedDataParallelTest testMethod=test_invalid_powerSGD_state>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_integer_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_torch_device_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_2gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_4gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_ids_not_allowed>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_empty_device_ids>, <__main__.DistributedDataParallelTest testMethod=test_nccl_propagate_error_reason>, <__main__.DistributedDataParallelTest testMethod=test_no_grad>, <__main__.DistributedDataParallelTest testMethod=test_param_layout_mismatch_error>, <__main__.DistributedDataParallelTest testMethod=test_pass_default_pg>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_empty_input>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_only_empty_input>]>
2022-11-23T02:25:23.0135743Z test_accumulate_gradients_module (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0136294Z test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0137824Z test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0138336Z test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0138787Z test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0139234Z test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0139676Z test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0140124Z test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0140555Z test_channels_last_contig (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0140994Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0141474Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0141942Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0142427Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0142942Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0143472Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0143966Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0144446Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0144931Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0145430Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0145926Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0146439Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0147058Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0147543Z test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0148016Z test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0148508Z test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0148987Z test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0149441Z test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0149895Z test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0150327Z test_ddp_packed_sequence (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0150728Z test_ddp_weight_sharing (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0151156Z test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0151594Z test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0152042Z test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0152540Z test_failure_recovery (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0152986Z test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0153468Z test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0153921Z test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0155723Z test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0156282Z test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0156826Z test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0157322Z test_fp16 (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0157793Z test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0158246Z test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0158725Z test_fp16_grad_is_view (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0159192Z test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0159726Z test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0160173Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0160662Z test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0161154Z test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0161777Z test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0162273Z test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0162826Z test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0163258Z test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0163776Z test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0164246Z test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0164859Z test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0165420Z test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0165878Z test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0166373Z test_no_grad (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0166912Z test_param_layout_mismatch_error (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0167354Z test_pass_default_pg (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0167825Z test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0168298Z test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0168815Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0169291Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0169685Z <unittest.suite.TestSuite tests=[]>
2022-11-23T02:25:23.0171062Z <unittest.suite.TestSuite tests=[<__main__.NcclErrorHandlingTest testMethod=test_invalid_nccl_blocking_wait_env>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_blocking_wait_with_barrier>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_abort>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_clean_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_nonzero_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigkill>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigterm>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_nonblocking>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_timeout>]>
2022-11-23T02:25:23.0172445Z test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest)
2022-11-23T02:25:23.0172944Z test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest)
2022-11-23T02:25:23.0173334Z test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest)
2022-11-23T02:25:23.0173812Z test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest)
2022-11-23T02:25:23.0174234Z test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest)
2022-11-23T02:25:23.0174694Z test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest)
2022-11-23T02:25:23.0175087Z test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest)
2022-11-23T02:25:23.0175531Z test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest)
2022-11-23T02:25:23.0175918Z test_nccl_timeout (__main__.NcclErrorHandlingTest)
2022-11-23T02:25:23.0177748Z <unittest.suite.TestSuite tests=[<__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_allgather_base>, <__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_allreduce_coalesced>, <__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_collectives>, <__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_reduce_scatter_base>]>
2022-11-23T02:25:23.0179053Z test_allgather_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests)
2022-11-23T02:25:23.0179644Z test_allreduce_coalesced (__main__.NcclProcessGroupWithDispatchedCollectivesTests)
2022-11-23T02:25:23.0180365Z test_collectives (__main__.NcclProcessGroupWithDispatchedCollectivesTests)
2022-11-23T02:25:23.0180973Z test_reduce_scatter_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests)
2022-11-23T02:25:23.0181517Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupNCCLNoGPUTest testMethod=test_init_no_gpus>]>
2022-11-23T02:25:23.0182060Z test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest)
2022-11-23T02:25:23.0184421Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupNCCLTest testMethod=test_allgather_base_basics>, <__main__.ProcessGroupNCCLTest testMethod=test_allgather_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allgather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allreduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_barrier>, <__main__.ProcessGroupNCCLTest testMethod=test_broadcast_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_empty_tensors>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_stress>, <__main__.ProcessGroupNCCLTest testMethod=test_nccl_dist_backend_error>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_basics>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_stress>, <__main__.ProcessGroupNCCLTest testMethod=test_send_recv>]>
2022-11-23T02:25:23.0186632Z test_allgather_base_basics (__main__.ProcessGroupNCCLTest)
2022-11-23T02:25:23.0187003Z test_allgather_base_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T02:25:23.0187377Z test_allgather_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T02:25:23.0187743Z test_allreduce_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T02:25:23.0188074Z test_barrier (__main__.ProcessGroupNCCLTest)
2022-11-23T02:25:23.0188429Z test_broadcast_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T02:25:23.0188793Z test_empty_tensors (__main__.ProcessGroupNCCLTest)
2022-11-23T02:25:23.0189127Z test_gather_checks (__main__.ProcessGroupNCCLTest)
2022-11-23T02:25:23.0189586Z test_gather_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T02:25:23.0189943Z test_gather_stress (__main__.ProcessGroupNCCLTest)
2022-11-23T02:25:23.0190311Z test_nccl_dist_backend_error (__main__.ProcessGroupNCCLTest)
2022-11-23T02:25:23.0190665Z test_reduce_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T02:25:23.0191042Z test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest)
2022-11-23T02:25:23.0191435Z test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T02:25:23.0191801Z test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T02:25:23.0192176Z test_scatter_checks (__main__.ProcessGroupNCCLTest)
2022-11-23T02:25:23.0192526Z test_scatter_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T02:25:23.0192872Z test_scatter_stress (__main__.ProcessGroupNCCLTest)
2022-11-23T02:25:23.0193230Z test_send_recv (__main__.ProcessGroupNCCLTest)
2022-11-23T02:25:23.0193657Z <unittest.suite.TestSuite tests=[<__main__.RendezvousEnvTest testMethod=test_common_errors>]>
2022-11-23T02:25:23.0194079Z test_common_errors (__main__.RendezvousEnvTest)
2022-11-23T02:25:23.0194394Z <unittest.suite.TestSuite tests=[]>
2022-11-23T02:25:23.0194818Z <unittest.suite.TestSuite tests=[<__main__.TimeoutTest testMethod=test_default_store_timeout_nccl>]>
2022-11-23T02:25:23.0195242Z test_default_store_timeout_nccl (__main__.TimeoutTest)
2022-11-23T02:25:23.0195952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0196419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0197003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0197480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0197699Z 
2022-11-23T02:25:23.0197811Z Running tests...
2022-11-23T02:25:23.0198223Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0198759Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0199223Z   test_all_reduce_coalesced_nccl (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0199686Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34432
2022-11-23T02:25:23.0200138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34433
2022-11-23T02:25:23.0200746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0204715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0205343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0205917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0206498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0206960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0207543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0208019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0208447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0208932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0209944Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:25:23.0210651Z   warnings.warn(
2022-11-23T02:25:23.0211508Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:25:23.0212135Z   warnings.warn(
2022-11-23T02:25:23.0212385Z ok (6.908s)
2022-11-23T02:25:23.0212538Z 
2022-11-23T02:25:23.0212815Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0213128Z Ran 1 test in 6.908s
2022-11-23T02:25:23.0213291Z 
2022-11-23T02:25:23.0213387Z OK
2022-11-23T02:25:23.0213523Z 
2022-11-23T02:25:23.0213650Z Generating XML reports...
2022-11-23T02:25:23.0214177Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020517.xml
2022-11-23T02:25:23.0214860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0215326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0215909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0216362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0216594Z 
2022-11-23T02:25:23.0217163Z Running tests...
2022-11-23T02:25:23.0217623Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0218141Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0218631Z   test_broadcast_coalesced_nccl (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0219101Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34645
2022-11-23T02:25:23.0219565Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34646
2022-11-23T02:25:23.0220164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0220622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0221198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0221654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0222236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0222679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0223246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0223804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0224271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0224759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0225090Z ok (6.923s)
2022-11-23T02:25:23.0225236Z 
2022-11-23T02:25:23.0225509Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0225843Z Ran 1 test in 6.923s
2022-11-23T02:25:23.0226007Z 
2022-11-23T02:25:23.0226103Z OK
2022-11-23T02:25:23.0226218Z 
2022-11-23T02:25:23.0226347Z Generating XML reports...
2022-11-23T02:25:23.0226897Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020526.xml
2022-11-23T02:25:23.0227563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0228000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0228586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0229146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0229377Z 
2022-11-23T02:25:23.0229482Z Running tests...
2022-11-23T02:25:23.0229870Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0230396Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0230850Z   test_nccl_barrier (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0231276Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34858
2022-11-23T02:25:23.0257936Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34859
2022-11-23T02:25:23.0258657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0259150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0259725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0260183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0260741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0261215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0261914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0262393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0262818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0263317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.0263806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0264271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.0264937Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0265630Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0266164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:25:23.0266644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:25:23.0267291Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:25:23.0268108Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:25:23.0268667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:25:23.0269143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:25:23.0269797Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:25:23.0270479Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:25:23.0271012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-11-23T02:25:23.0271490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-11-23T02:25:23.0272140Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-11-23T02:25:23.0272903Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-11-23T02:25:23.0273275Z ok (7.069s)
2022-11-23T02:25:23.0273428Z 
2022-11-23T02:25:23.0273700Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0274027Z Ran 1 test in 7.070s
2022-11-23T02:25:23.0274190Z 
2022-11-23T02:25:23.0274284Z OK
2022-11-23T02:25:23.0274400Z 
2022-11-23T02:25:23.0274526Z Generating XML reports...
2022-11-23T02:25:23.0275074Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020536.xml
2022-11-23T02:25:23.0275741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0276176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0276756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0277230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0277458Z 
2022-11-23T02:25:23.0277567Z Running tests...
2022-11-23T02:25:23.0277957Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0278484Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0278958Z   test_nccl_barrier_device_ids (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0279402Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35094
2022-11-23T02:25:23.0279857Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35095
2022-11-23T02:25:23.0280458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0280911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0281471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0281939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0282515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0282941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0283515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0283977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0284412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0284937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.0285428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0285921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.0286579Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0287257Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0287648Z ok (5.728s)
2022-11-23T02:25:23.0287796Z 
2022-11-23T02:25:23.0288064Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0288374Z Ran 1 test in 5.728s
2022-11-23T02:25:23.0288538Z 
2022-11-23T02:25:23.0288629Z OK
2022-11-23T02:25:23.0288763Z 
2022-11-23T02:25:23.0288885Z Generating XML reports...
2022-11-23T02:25:23.0289432Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020545.xml
2022-11-23T02:25:23.0290084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0290592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0291167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0291625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0291858Z 
2022-11-23T02:25:23.0291967Z Running tests...
2022-11-23T02:25:23.0292373Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0292907Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0293396Z   test_nccl_barrier_device_ids_function_argument (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0293885Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35306
2022-11-23T02:25:23.0294356Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35307
2022-11-23T02:25:23.0294960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0295414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0295994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0296463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0297260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0297713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0298285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0298751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0299178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0299673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.0300160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0300621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.0301278Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0301968Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0302359Z ok (4.105s)
2022-11-23T02:25:23.0302490Z 
2022-11-23T02:25:23.0302837Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0303172Z Ran 1 test in 4.105s
2022-11-23T02:25:23.0303337Z 
2022-11-23T02:25:23.0303430Z OK
2022-11-23T02:25:23.0303563Z 
2022-11-23T02:25:23.0303670Z Generating XML reports...
2022-11-23T02:25:23.0304217Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020553.xml
2022-11-23T02:25:23.0304884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0305333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0305888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0306354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0306586Z 
2022-11-23T02:25:23.0306694Z Running tests...
2022-11-23T02:25:23.0307082Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0307615Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0308153Z   test_nccl_barrier_timeout (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0308611Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35509
2022-11-23T02:25:23.0309047Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35510
2022-11-23T02:25:23.0309652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0310106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0310664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0311134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0311714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0312163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0312713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0313175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0313615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0314085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0314413Z ok (14.135s)
2022-11-23T02:25:23.0314563Z 
2022-11-23T02:25:23.0314837Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0315166Z Ran 1 test in 14.136s
2022-11-23T02:25:23.0315328Z 
2022-11-23T02:25:23.0315402Z OK
2022-11-23T02:25:23.0315534Z 
2022-11-23T02:25:23.0315661Z Generating XML reports...
2022-11-23T02:25:23.0316208Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020600.xml
2022-11-23T02:25:23.0316859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0317311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0317886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0318355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0318585Z 
2022-11-23T02:25:23.0318676Z Running tests...
2022-11-23T02:25:23.0319074Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0319602Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0320141Z   test_nccl_barrier_timeout_new_group (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0320603Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35712
2022-11-23T02:25:23.0321065Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35713
2022-11-23T02:25:23.0321675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0322108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0322682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0323148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0323725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0324153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0324726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0325264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0325685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0326159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0326645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.0327148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.0327787Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0328481Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0329014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:25:23.0329370Z ok (9.117s)
2022-11-23T02:25:23.0329500Z 
2022-11-23T02:25:23.0329768Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0330093Z Ran 1 test in 9.117s
2022-11-23T02:25:23.0330256Z 
2022-11-23T02:25:23.0330347Z OK
2022-11-23T02:25:23.0330481Z 
2022-11-23T02:25:23.0330588Z Generating XML reports...
2022-11-23T02:25:23.0331133Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020617.xml
2022-11-23T02:25:23.0331796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0332246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0332809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0333279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0333516Z 
2022-11-23T02:25:23.0333625Z Running tests...
2022-11-23T02:25:23.0334012Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0334538Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0335044Z   test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0335529Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35936
2022-11-23T02:25:23.0335964Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35937
2022-11-23T02:25:23.0336570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0337311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0338002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0338488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0339072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0339516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0340069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0340530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0340970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0341425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0341920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.0342502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.0343231Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0343907Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0344440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:25:23.0344794Z ok (9.035s)
2022-11-23T02:25:23.0344944Z 
2022-11-23T02:25:23.0345209Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0345518Z Ran 1 test in 9.035s
2022-11-23T02:25:23.0345681Z 
2022-11-23T02:25:23.0345773Z OK
2022-11-23T02:25:23.0345906Z 
2022-11-23T02:25:23.0346030Z Generating XML reports...
2022-11-23T02:25:23.0346559Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020628.xml
2022-11-23T02:25:23.0347227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0347676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0348252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0348706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0348937Z 
2022-11-23T02:25:23.0349044Z Running tests...
2022-11-23T02:25:23.0349447Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0349959Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0350449Z   test_nccl_warn_not_in_group_debug_detail (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0350930Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36151
2022-11-23T02:25:23.0351388Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36152
2022-11-23T02:25:23.0351981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0352430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0352995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0353423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0353998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0354469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0355151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0355609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0356052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0356537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0357029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.0357512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.0358183Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0358886Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0359412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:25:23.0359975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:25:23.0360641Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:25:23.0361338Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:25:23.0361814Z ok (5.772s)
2022-11-23T02:25:23.0361976Z 
2022-11-23T02:25:23.0362260Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0362596Z Ran 1 test in 5.772s
2022-11-23T02:25:23.0362763Z 
2022-11-23T02:25:23.0362860Z OK
2022-11-23T02:25:23.0362976Z 
2022-11-23T02:25:23.0363105Z Generating XML reports...
2022-11-23T02:25:23.0363657Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020640.xml
2022-11-23T02:25:23.0364338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0364781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0365368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0365850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0366085Z 
2022-11-23T02:25:23.0366198Z Running tests...
2022-11-23T02:25:23.0366591Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0367127Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0367628Z   test_nccl_warn_not_in_group_debug_info (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0368090Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36378
2022-11-23T02:25:23.0368554Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36379
2022-11-23T02:25:23.0369178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0369639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0370200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0370683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0371274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0371703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0372285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0372817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0373284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0373765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.0374257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0374753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.0375396Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0376102Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0376918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:25:23.0377445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:25:23.0378191Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:25:23.0378888Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:25:23.0379299Z ok (5.764s)
2022-11-23T02:25:23.0379456Z 
2022-11-23T02:25:23.0379731Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0380044Z Ran 1 test in 5.764s
2022-11-23T02:25:23.0380210Z 
2022-11-23T02:25:23.0380308Z OK
2022-11-23T02:25:23.0380446Z 
2022-11-23T02:25:23.0380576Z Generating XML reports...
2022-11-23T02:25:23.0381108Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020648.xml
2022-11-23T02:25:23.0381792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0382256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0382844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0383302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0383535Z 
2022-11-23T02:25:23.0383648Z Running tests...
2022-11-23T02:25:23.0384063Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0384579Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0385076Z   test_nccl_warn_not_in_group_debug_off (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0385562Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36596
2022-11-23T02:25:23.0386024Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36597
2022-11-23T02:25:23.0386626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0387093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0387673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0388155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0388720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0389175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0389750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0390205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0390724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0391232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.0391727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0392198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.0392873Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0393421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:25:23.0394084Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0394602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:25:23.0395262Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:25:23.0396016Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:25:23.0396394Z ok (5.818s)
2022-11-23T02:25:23.0396552Z 
2022-11-23T02:25:23.0396826Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0397169Z Ran 1 test in 5.818s
2022-11-23T02:25:23.0397336Z 
2022-11-23T02:25:23.0397434Z OK
2022-11-23T02:25:23.0397550Z 
2022-11-23T02:25:23.0397679Z Generating XML reports...
2022-11-23T02:25:23.0398233Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020656.xml
2022-11-23T02:25:23.0398909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0399353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0399942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0400428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0400666Z 
2022-11-23T02:25:23.0400779Z Running tests...
2022-11-23T02:25:23.0401170Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0401709Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0402189Z   test_nncl_rank_membership (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0402641Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36814
2022-11-23T02:25:23.0403106Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36815
2022-11-23T02:25:23.0403733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0404193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0404762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0405239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0405827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0406256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0406834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0407311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0407757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0408294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.0408801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0409301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.0409966Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0410487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:25:23.0411138Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0411666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:25:23.0412290Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:25:23.0412983Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:25:23.0413458Z ok (3.963s)
2022-11-23T02:25:23.0413618Z 
2022-11-23T02:25:23.0413892Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0414205Z Ran 1 test in 3.964s
2022-11-23T02:25:23.0414369Z 
2022-11-23T02:25:23.0414468Z OK
2022-11-23T02:25:23.0414606Z 
2022-11-23T02:25:23.0414735Z Generating XML reports...
2022-11-23T02:25:23.0415268Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020704.xml
2022-11-23T02:25:23.0415947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0416416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0417297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0417759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0418053Z 
2022-11-23T02:25:23.0418172Z Running tests...
2022-11-23T02:25:23.0418592Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0419137Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0419624Z   test_pass_nccl_options_high_priority_stream (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0420116Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37019
2022-11-23T02:25:23.0420584Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37020
2022-11-23T02:25:23.0421182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0421639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0422234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0422720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0423289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0423742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0424323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0424776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0425230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0425739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.0426321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0426801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.0427478Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0428023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:25:23.0428683Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0429204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:25:23.0429857Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:25:23.0430555Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:25:23.0430934Z ok (6.773s)
2022-11-23T02:25:23.0431157Z 
2022-11-23T02:25:23.0431434Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0431770Z Ran 1 test in 6.773s
2022-11-23T02:25:23.0431939Z 
2022-11-23T02:25:23.0432036Z OK
2022-11-23T02:25:23.0432150Z 
2022-11-23T02:25:23.0432280Z Generating XML reports...
2022-11-23T02:25:23.0432838Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020711.xml
2022-11-23T02:25:23.0433512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0433952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0434534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0435014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0435252Z 
2022-11-23T02:25:23.0435367Z Running tests...
2022-11-23T02:25:23.0435767Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0436304Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0436815Z   test_sequence_num_incremented_nccl_default (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0437308Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37236
2022-11-23T02:25:23.0437748Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37237
2022-11-23T02:25:23.0438373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0438835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0439398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0439881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0440480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0440936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0441495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0441975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0442424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0442906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.0443402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0443948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.0444635Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0445305Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0445847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:25:23.0446351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:25:23.0447010Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:25:23.0447682Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:25:23.0448084Z ok (5.772s)
2022-11-23T02:25:23.0448245Z 
2022-11-23T02:25:23.0448519Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0448903Z Ran 1 test in 5.772s
2022-11-23T02:25:23.0449073Z 
2022-11-23T02:25:23.0449171Z OK
2022-11-23T02:25:23.0449308Z 
2022-11-23T02:25:23.0449436Z Generating XML reports...
2022-11-23T02:25:23.0449997Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020720.xml
2022-11-23T02:25:23.0450652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0451112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0451699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0452158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0452399Z 
2022-11-23T02:25:23.0452510Z Running tests...
2022-11-23T02:25:23.0452927Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0453470Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0453963Z   test_sequence_num_incremented_nccl_subgroup (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0454457Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37454
2022-11-23T02:25:23.0454920Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37455
2022-11-23T02:25:23.0455519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0455981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0456570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0457240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0457823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0458284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0458862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0459330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0459755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0460241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0460596Z ok (3.995s)
2022-11-23T02:25:23.0460750Z 
2022-11-23T02:25:23.0461005Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0461341Z Ran 1 test in 3.995s
2022-11-23T02:25:23.0461507Z 
2022-11-23T02:25:23.0461752Z OK
2022-11-23T02:25:23.0461912Z 
2022-11-23T02:25:23.0462020Z Generating XML reports...
2022-11-23T02:25:23.0462591Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020728.xml
2022-11-23T02:25:23.0463265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0463723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0464281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0464754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0464990Z 
2022-11-23T02:25:23.0465101Z Running tests...
2022-11-23T02:25:23.0465518Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0466038Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0466546Z   test_sequence_num_set_default_pg_nccl (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0467099Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37653
2022-11-23T02:25:23.0467538Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37654
2022-11-23T02:25:23.0468181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0468639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0469211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0469647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0470226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0470708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0471285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0471769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0472222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0472725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.0473194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0473687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.0474357Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0475066Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0475454Z ok (5.677s)
2022-11-23T02:25:23.0475606Z 
2022-11-23T02:25:23.0475883Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0476217Z Ran 1 test in 5.677s
2022-11-23T02:25:23.0476385Z 
2022-11-23T02:25:23.0476459Z OK
2022-11-23T02:25:23.0476598Z 
2022-11-23T02:25:23.0476729Z Generating XML reports...
2022-11-23T02:25:23.0477278Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020735.xml
2022-11-23T02:25:23.0477955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0478389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0478970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0479502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0479747Z 
2022-11-23T02:25:23.0479845Z Running tests...
2022-11-23T02:25:23.0480261Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0480800Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0481298Z   test_sequence_num_set_nccl_new_group (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0481756Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37865
2022-11-23T02:25:23.0482217Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37866
2022-11-23T02:25:23.0482834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0483262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0483853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0484404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0485057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0485487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0486060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0486532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0486956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0487454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.0487954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0488451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.0489099Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0489646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:25:23.0490304Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0490848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:25:23.0491489Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:25:23.0492178Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:25:23.0492584Z ok (5.795s)
2022-11-23T02:25:23.0492739Z 
2022-11-23T02:25:23.0493014Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0493330Z Ran 1 test in 5.795s
2022-11-23T02:25:23.0493504Z 
2022-11-23T02:25:23.0493599Z OK
2022-11-23T02:25:23.0493737Z 
2022-11-23T02:25:23.0493869Z Generating XML reports...
2022-11-23T02:25:23.0494400Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020743.xml
2022-11-23T02:25:23.0495082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0495545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0496104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0496582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0497062Z 
2022-11-23T02:25:23.0497255Z Running tests...
2022-11-23T02:25:23.0497687Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0498213Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0498701Z   test_tensor_dtype_complex (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0499170Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38081
2022-11-23T02:25:23.0499614Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38082
2022-11-23T02:25:23.0500240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0500701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0501273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0501731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0502309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0502822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0503396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0503845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0504282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0504775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.0505240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0505718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.0506497Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0507190Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0507565Z ok (6.922s)
2022-11-23T02:25:23.0507713Z 
2022-11-23T02:25:23.0507982Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0508305Z Ran 1 test in 6.923s
2022-11-23T02:25:23.0508467Z 
2022-11-23T02:25:23.0508540Z OK
2022-11-23T02:25:23.0508674Z 
2022-11-23T02:25:23.0508799Z Generating XML reports...
2022-11-23T02:25:23.0509342Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020751.xml
2022-11-23T02:25:23.0510007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0510443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0511018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0511491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0511725Z 
2022-11-23T02:25:23.0511834Z Running tests...
2022-11-23T02:25:23.0512316Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0512842Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0513321Z   test_tensor_dtype_mismatch (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0513761Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38294
2022-11-23T02:25:23.0514213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38295
2022-11-23T02:25:23.0514875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0515332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0515894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0516360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0516935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0517361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0517929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0518484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0518921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0519401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.0519881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0520427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.0521066Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0521756Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0522809Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:25:23.0523460Z   warnings.warn(
2022-11-23T02:25:23.0524332Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:25:23.0524935Z   warnings.warn(
2022-11-23T02:25:23.0525800Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:25:23.0526415Z   warnings.warn(
2022-11-23T02:25:23.0527282Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:25:23.0527879Z   warnings.warn(
2022-11-23T02:25:23.0528117Z ok (5.633s)
2022-11-23T02:25:23.0528267Z 
2022-11-23T02:25:23.0528538Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0528845Z Ran 1 test in 5.633s
2022-11-23T02:25:23.0529009Z 
2022-11-23T02:25:23.0529101Z OK
2022-11-23T02:25:23.0529233Z 
2022-11-23T02:25:23.0529358Z Generating XML reports...
2022-11-23T02:25:23.0529898Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020801.xml
2022-11-23T02:25:23.0530542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0530990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0531580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0532104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0532346Z 
2022-11-23T02:25:23.0532459Z Running tests...
2022-11-23T02:25:23.0532878Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0533414Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0533887Z   test_allgather_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0534364Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38499
2022-11-23T02:25:23.0534824Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38500
2022-11-23T02:25:23.0535418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0535870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0536447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0537140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0537798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0538248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0538820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0539283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0539705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0540174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0540664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.0541150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.0541810Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0542502Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0543423Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:25:23.0544132Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:25:23.0544990Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:25:23.0545701Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:25:23.0546033Z ok (7.042s)
2022-11-23T02:25:23.0546180Z 
2022-11-23T02:25:23.0546430Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0546765Z Ran 1 test in 7.042s
2022-11-23T02:25:23.0546928Z 
2022-11-23T02:25:23.0547019Z OK
2022-11-23T02:25:23.0547155Z 
2022-11-23T02:25:23.0547277Z Generating XML reports...
2022-11-23T02:25:23.0547816Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123020809.xml
2022-11-23T02:25:23.0548487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0548933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0549556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0550043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0550278Z 
2022-11-23T02:25:23.0550384Z Running tests...
2022-11-23T02:25:23.0550790Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0551296Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0551778Z   test_allreduce_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0552249Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38710
2022-11-23T02:25:23.0552682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38711
2022-11-23T02:25:23.0553300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0553749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0554336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0554859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0555449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0555903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0556485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0556937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0557392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0557873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0558351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.0558860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.0559533Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0560236Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0561142Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:25:23.0561969Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:25:23.0562846Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:25:23.0563570Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:25:23.0563884Z ok (6.945s)
2022-11-23T02:25:23.0564037Z 
2022-11-23T02:25:23.0564310Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0564644Z Ran 1 test in 6.945s
2022-11-23T02:25:23.0564810Z 
2022-11-23T02:25:23.0564903Z OK
2022-11-23T02:25:23.0565018Z 
2022-11-23T02:25:23.0565142Z Generating XML reports...
2022-11-23T02:25:23.0565709Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123020818.xml
2022-11-23T02:25:23.0566391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0566826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0567472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0567975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0568211Z 
2022-11-23T02:25:23.0568324Z Running tests...
2022-11-23T02:25:23.0568718Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0569251Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0569751Z   test_broadcast_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0570211Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38921
2022-11-23T02:25:23.0570676Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38922
2022-11-23T02:25:23.0571293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0571754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0572391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0572867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0573459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0573890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0574468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0574944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0575394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0575854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0576349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.0577106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.0577793Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0578478Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0579412Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:25:23.0580137Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:25:23.0581003Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:25:23.0581708Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:25:23.0582047Z ok (6.918s)
2022-11-23T02:25:23.0582200Z 
2022-11-23T02:25:23.0582474Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0582811Z Ran 1 test in 6.918s
2022-11-23T02:25:23.0582955Z 
2022-11-23T02:25:23.0583059Z OK
2022-11-23T02:25:23.0583198Z 
2022-11-23T02:25:23.0583326Z Generating XML reports...
2022-11-23T02:25:23.0583894Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123020828.xml
2022-11-23T02:25:23.0584555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0585087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0585695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0586179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0586391Z 
2022-11-23T02:25:23.0586509Z Running tests...
2022-11-23T02:25:23.0586924Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0587464Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0587951Z   test_consecutive_comm_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0588442Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39132
2022-11-23T02:25:23.0588907Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39133
2022-11-23T02:25:23.0589532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0590045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0590634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0591109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0591671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0592192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0592771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0593243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0593676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0594162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0594668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.0595174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.0595819Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0596517Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0597454Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:25:23.0598189Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:25:23.0599030Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:25:23.0599755Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:25:23.0600616Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:25:23.0601332Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:25:23.0602218Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:25:23.0602950Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:25:23.0603812Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:25:23.0604532Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:25:23.0605392Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:25:23.0606092Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:25:23.0606949Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:25:23.0607722Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:25:23.0608573Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:25:23.0609271Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:25:23.0609606Z ok (7.007s)
2022-11-23T02:25:23.0609757Z 
2022-11-23T02:25:23.0610035Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0610374Z Ran 1 test in 7.007s
2022-11-23T02:25:23.0610521Z 
2022-11-23T02:25:23.0610620Z OK
2022-11-23T02:25:23.0610762Z 
2022-11-23T02:25:23.0610896Z Generating XML reports...
2022-11-23T02:25:23.0611463Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123020837.xml
2022-11-23T02:25:23.0612129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0612586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0613170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0613650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0613862Z 
2022-11-23T02:25:23.0613975Z Running tests...
2022-11-23T02:25:23.0614385Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0614928Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0615409Z   test_nested_comm_tensor_wrapping (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0615901Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39343
2022-11-23T02:25:23.0616365Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39344
2022-11-23T02:25:23.0617192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0617636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0618225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0618703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0619345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0619816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0620407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0620878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0621300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0621780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0622283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.0622790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.0623436Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0624137Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0625183Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:25:23.0625913Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:25:23.0626750Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:25:23.0627474Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:25:23.0628335Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:25:23.0629064Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:25:23.0629902Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:25:23.0630620Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:25:23.0630953Z ok (6.919s)
2022-11-23T02:25:23.0631104Z 
2022-11-23T02:25:23.0631377Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0631695Z Ran 1 test in 6.919s
2022-11-23T02:25:23.0631865Z 
2022-11-23T02:25:23.0631964Z OK
2022-11-23T02:25:23.0632101Z 
2022-11-23T02:25:23.0632234Z Generating XML reports...
2022-11-23T02:25:23.0632778Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123020847.xml
2022-11-23T02:25:23.0633458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0633922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0634508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0634966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0635208Z 
2022-11-23T02:25:23.0635323Z Running tests...
2022-11-23T02:25:23.0635741Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0636275Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0636806Z   test_reduce_scatter_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0637301Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39554
2022-11-23T02:25:23.0637766Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39555
2022-11-23T02:25:23.0638368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0638835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0639414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0639892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0640462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0640924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0641506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0642014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0642461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0642946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0643442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.0643928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.0644600Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0645313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0646247Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:25:23.0646960Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:25:23.0647822Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:25:23.0648545Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:25:23.0649409Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:25:23.0650135Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:25:23.0650972Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:25:23.0651686Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:25:23.0652026Z ok (6.929s)
2022-11-23T02:25:23.0652178Z 
2022-11-23T02:25:23.0652428Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0652765Z Ran 1 test in 6.929s
2022-11-23T02:25:23.0652932Z 
2022-11-23T02:25:23.0653029Z OK
2022-11-23T02:25:23.0653197Z 
2022-11-23T02:25:23.0653326Z Generating XML reports...
2022-11-23T02:25:23.0653922Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123020856.xml
2022-11-23T02:25:23.0654617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0655080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0655641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0656120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0656361Z 
2022-11-23T02:25:23.0656474Z Running tests...
2022-11-23T02:25:23.0657113Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0657644Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0658137Z   test_scatter_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0658619Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39765
2022-11-23T02:25:23.0659059Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39766
2022-11-23T02:25:23.0659770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0660238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0660820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0661282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0661963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0662428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0663019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0663474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0663927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0664405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0664880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.0665393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.0666067Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0666766Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0667686Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:25:23.0668424Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:25:23.0669282Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-11-23T02:25:23.0670004Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-11-23T02:25:23.0670320Z ok (6.908s)
2022-11-23T02:25:23.0670475Z 
2022-11-23T02:25:23.0670750Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0671094Z Ran 1 test in 6.908s
2022-11-23T02:25:23.0671260Z 
2022-11-23T02:25:23.0671358Z OK
2022-11-23T02:25:23.0671474Z 
2022-11-23T02:25:23.0671675Z Generating XML reports...
2022-11-23T02:25:23.0672253Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123020905.xml
2022-11-23T02:25:23.0672943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0673375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0673961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0674436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0674667Z 
2022-11-23T02:25:23.0674779Z Running tests...
2022-11-23T02:25:23.0675170Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0675704Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0676257Z   test_accumulate_gradients_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0676840Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39978
2022-11-23T02:25:23.0677358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39979
2022-11-23T02:25:23.0677981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0678441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0679009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0679485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0680079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0680531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0681097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0681578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0682028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0682488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0682759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn7t5hdn8
2022-11-23T02:25:23.0683036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn7t5hdn8/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0683297Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4co4vham
2022-11-23T02:25:23.0683575Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4co4vham/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0683820Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0684063Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0684173Z ok (7.477s)
2022-11-23T02:25:23.0684193Z 
2022-11-23T02:25:23.0684446Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0684568Z Ran 1 test in 7.477s
2022-11-23T02:25:23.0684587Z 
2022-11-23T02:25:23.0684680Z OK
2022-11-23T02:25:23.0684699Z 
2022-11-23T02:25:23.0684829Z Generating XML reports...
2022-11-23T02:25:23.0685303Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123020915.xml
2022-11-23T02:25:23.0685684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0685864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0686299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0686507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0686527Z 
2022-11-23T02:25:23.0686617Z Running tests...
2022-11-23T02:25:23.0686891Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0687209Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0687524Z   test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0687751Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40199
2022-11-23T02:25:23.0687977Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40200
2022-11-23T02:25:23.0688353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0688538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0688902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0689178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0689556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0689737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0690111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0690307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0690547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0690783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0691051Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk0icdkxi
2022-11-23T02:25:23.0691310Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk0icdkxi/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0691572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphguax3gc
2022-11-23T02:25:23.0691846Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphguax3gc/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0692089Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0692333Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0692438Z ok (7.489s)
2022-11-23T02:25:23.0692457Z 
2022-11-23T02:25:23.0692732Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0692847Z Ran 1 test in 7.490s
2022-11-23T02:25:23.0692866Z 
2022-11-23T02:25:23.0692943Z OK
2022-11-23T02:25:23.0692962Z 
2022-11-23T02:25:23.0693091Z Generating XML reports...
2022-11-23T02:25:23.0693566Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123020925.xml
2022-11-23T02:25:23.0693945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0694133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0694517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0694717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0694737Z 
2022-11-23T02:25:23.0694850Z Running tests...
2022-11-23T02:25:23.0695118Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0695554Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0695855Z   test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0696086Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40420
2022-11-23T02:25:23.0696315Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40421
2022-11-23T02:25:23.0698546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0698737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0699216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0699400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0699766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0699970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0700483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0700684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0700921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0701154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0701419Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2nhvsm45
2022-11-23T02:25:23.0701694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2nhvsm45/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0701956Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy7go1glk
2022-11-23T02:25:23.0702212Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy7go1glk/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0702455Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0702700Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0702806Z ok (7.403s)
2022-11-23T02:25:23.0702827Z 
2022-11-23T02:25:23.0703105Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0703221Z Ran 1 test in 7.404s
2022-11-23T02:25:23.0703241Z 
2022-11-23T02:25:23.0703334Z OK
2022-11-23T02:25:23.0703354Z 
2022-11-23T02:25:23.0703484Z Generating XML reports...
2022-11-23T02:25:23.0703927Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123020935.xml
2022-11-23T02:25:23.0704305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0704486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0704872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0705073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0705093Z 
2022-11-23T02:25:23.0705205Z Running tests...
2022-11-23T02:25:23.0705474Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0705794Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0706101Z   test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0706307Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40641
2022-11-23T02:25:23.0706594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40642
2022-11-23T02:25:23.0707047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0707243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0707631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0707828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0708201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0708382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0708736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0708929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0709166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0709402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0709719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzfzr22_n
2022-11-23T02:25:23.0709996Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzfzr22_n/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0710255Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpswupgbh9
2022-11-23T02:25:23.0710528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpswupgbh9/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0710774Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0710985Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0711094Z ok (7.522s)
2022-11-23T02:25:23.0711114Z 
2022-11-23T02:25:23.0711397Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0711516Z Ran 1 test in 7.522s
2022-11-23T02:25:23.0711535Z 
2022-11-23T02:25:23.0711634Z OK
2022-11-23T02:25:23.0711653Z 
2022-11-23T02:25:23.0711782Z Generating XML reports...
2022-11-23T02:25:23.0712252Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123020945.xml
2022-11-23T02:25:23.0712631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0712789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0713173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0713366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0713386Z 
2022-11-23T02:25:23.0713501Z Running tests...
2022-11-23T02:25:23.0713774Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0714094Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0714385Z   test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0714608Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40862
2022-11-23T02:25:23.0714832Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40863
2022-11-23T02:25:23.0715186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0715367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0715749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0715946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0716366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0716560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0716950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0717144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0717355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0717910Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.0718213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0718755Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.0719075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7lri60_4
2022-11-23T02:25:23.0719348Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7lri60_4/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0719610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq7epmib6
2022-11-23T02:25:23.0719885Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq7epmib6/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0719993Z ok (6.925s)
2022-11-23T02:25:23.0720013Z 
2022-11-23T02:25:23.0720296Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0720413Z Ran 1 test in 6.925s
2022-11-23T02:25:23.0720435Z 
2022-11-23T02:25:23.0720508Z OK
2022-11-23T02:25:23.0720528Z 
2022-11-23T02:25:23.0720657Z Generating XML reports...
2022-11-23T02:25:23.0721128Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123020955.xml
2022-11-23T02:25:23.0721504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0721684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0722069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0722269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0722289Z 
2022-11-23T02:25:23.0722401Z Running tests...
2022-11-23T02:25:23.0722650Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0722971Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0723262Z   test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0723492Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41083
2022-11-23T02:25:23.0723718Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41084
2022-11-23T02:25:23.0724093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0724273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0724656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0724850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0725252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0725445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0725833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0726029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0726266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0726814Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.0727051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0727591Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.0727904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6clr9i6a
2022-11-23T02:25:23.0728183Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6clr9i6a/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0728419Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfyb00ocs
2022-11-23T02:25:23.0728697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfyb00ocs/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0728806Z ok (7.016s)
2022-11-23T02:25:23.0728825Z 
2022-11-23T02:25:23.0729102Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0729223Z Ran 1 test in 7.016s
2022-11-23T02:25:23.0729243Z 
2022-11-23T02:25:23.0729339Z OK
2022-11-23T02:25:23.0729358Z 
2022-11-23T02:25:23.0729489Z Generating XML reports...
2022-11-23T02:25:23.0729958Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021004.xml
2022-11-23T02:25:23.0730335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0730495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0730884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0731084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0731104Z 
2022-11-23T02:25:23.0731216Z Running tests...
2022-11-23T02:25:23.0731493Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0731817Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0732102Z   test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0732328Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41304
2022-11-23T02:25:23.0732528Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41305
2022-11-23T02:25:23.0732905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0733087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0733475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0733718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0734100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0734285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0734666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0734859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0735071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0735309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0735574Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm983bmbn
2022-11-23T02:25:23.0735849Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm983bmbn/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0736114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprpozgk6u
2022-11-23T02:25:23.0736442Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprpozgk6u/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0736549Z ok (6.929s)
2022-11-23T02:25:23.0736569Z 
2022-11-23T02:25:23.0737102Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0737200Z Ran 1 test in 6.929s
2022-11-23T02:25:23.0737243Z 
2022-11-23T02:25:23.0737317Z OK
2022-11-23T02:25:23.0737336Z 
2022-11-23T02:25:23.0737463Z Generating XML reports...
2022-11-23T02:25:23.0737938Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021014.xml
2022-11-23T02:25:23.0738316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0738498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0738885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0739086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0739106Z 
2022-11-23T02:25:23.0739218Z Running tests...
2022-11-23T02:25:23.0739464Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0739780Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0740082Z   test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0740317Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41525
2022-11-23T02:25:23.0740545Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41526
2022-11-23T02:25:23.0740922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0741102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0741488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0741665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0742032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0742209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0742586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0742784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0743020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0743331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0743603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqgcrjjj4
2022-11-23T02:25:23.0743884Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqgcrjjj4/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0744121Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpywnskf9l
2022-11-23T02:25:23.0744396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpywnskf9l/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0744502Z ok (6.925s)
2022-11-23T02:25:23.0744522Z 
2022-11-23T02:25:23.0744797Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0744910Z Ran 1 test in 6.926s
2022-11-23T02:25:23.0744929Z 
2022-11-23T02:25:23.0745031Z OK
2022-11-23T02:25:23.0745050Z 
2022-11-23T02:25:23.0745178Z Generating XML reports...
2022-11-23T02:25:23.0745648Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021023.xml
2022-11-23T02:25:23.0745998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0746250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0746639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0746835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0746855Z 
2022-11-23T02:25:23.0746969Z Running tests...
2022-11-23T02:25:23.0747235Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0747551Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0747828Z   test_channels_last_contig (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0748061Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41746
2022-11-23T02:25:23.0748264Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41747
2022-11-23T02:25:23.0748642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0748821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0749207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0749401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0749767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0749947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0750329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0750502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0750740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0750973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0751080Z ok (6.998s)
2022-11-23T02:25:23.0751099Z 
2022-11-23T02:25:23.0751370Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0751483Z Ran 1 test in 6.998s
2022-11-23T02:25:23.0751503Z 
2022-11-23T02:25:23.0751600Z OK
2022-11-23T02:25:23.0751619Z 
2022-11-23T02:25:23.0751747Z Generating XML reports...
2022-11-23T02:25:23.0752213Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021032.xml
2022-11-23T02:25:23.0752621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0752809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0753194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0753389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0753408Z 
2022-11-23T02:25:23.0753521Z Running tests...
2022-11-23T02:25:23.0753787Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0754104Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0754339Z   test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0754687Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0754912Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41959
2022-11-23T02:25:23.0755135Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41960
2022-11-23T02:25:23.0755573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0755754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0756133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0756332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0756706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0756888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0757238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0757438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0757674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0757912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0758175Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpykvbwu7f
2022-11-23T02:25:23.0758454Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpykvbwu7f/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0758708Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp3n9n79b
2022-11-23T02:25:23.0758974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp3n9n79b/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0759058Z ok (6.289s)
2022-11-23T02:25:23.0759077Z 
2022-11-23T02:25:23.0759346Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0759456Z Ran 1 test in 6.289s
2022-11-23T02:25:23.0759479Z 
2022-11-23T02:25:23.0759570Z OK
2022-11-23T02:25:23.0759589Z 
2022-11-23T02:25:23.0759714Z Generating XML reports...
2022-11-23T02:25:23.0760178Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021042.xml
2022-11-23T02:25:23.0760551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0760730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0761109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0761283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0761304Z 
2022-11-23T02:25:23.0761418Z Running tests...
2022-11-23T02:25:23.0761689Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0762153Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0762408Z   test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0762687Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0762911Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42179
2022-11-23T02:25:23.0763134Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42180
2022-11-23T02:25:23.0763495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0763676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0764060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0764258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0764632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0764864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0765247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0765444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0765682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0765894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0766157Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgidaaqjb
2022-11-23T02:25:23.0766436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgidaaqjb/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0766699Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4zkiseck
2022-11-23T02:25:23.0766974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4zkiseck/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0767083Z ok (6.276s)
2022-11-23T02:25:23.0767103Z 
2022-11-23T02:25:23.0767378Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0767494Z Ran 1 test in 6.276s
2022-11-23T02:25:23.0767514Z 
2022-11-23T02:25:23.0767587Z OK
2022-11-23T02:25:23.0767605Z 
2022-11-23T02:25:23.0767737Z Generating XML reports...
2022-11-23T02:25:23.0768209Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021051.xml
2022-11-23T02:25:23.0768584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0768765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0769150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0769365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0769384Z 
2022-11-23T02:25:23.0769497Z Running tests...
2022-11-23T02:25:23.0769764Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0770058Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0770307Z   test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0770564Z DDP works as expected when layer is checkpointed only once. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0770791Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42399
2022-11-23T02:25:23.0771016Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42400
2022-11-23T02:25:23.0771437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0771623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0772005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0772178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0772539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0772714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0773088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0773276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0773507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0773742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0774050Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp012s48j7
2022-11-23T02:25:23.0774300Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp012s48j7/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0774557Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplgqfj4py
2022-11-23T02:25:23.0774828Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplgqfj4py/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0775065Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0775301Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0775530Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0775769Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0776680Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:25:23.0776987Z   warnings.warn(
2022-11-23T02:25:23.0777903Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:25:23.0778016Z   warnings.warn(
2022-11-23T02:25:23.0778240Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0778473Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0778707Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0778939Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0779173Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0779399Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0779623Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0779849Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0779932Z ok (6.530s)
2022-11-23T02:25:23.0779952Z 
2022-11-23T02:25:23.0780299Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0780420Z Ran 1 test in 6.530s
2022-11-23T02:25:23.0780443Z 
2022-11-23T02:25:23.0780534Z OK
2022-11-23T02:25:23.0780553Z 
2022-11-23T02:25:23.0780678Z Generating XML reports...
2022-11-23T02:25:23.0781145Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021059.xml
2022-11-23T02:25:23.0781516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0781693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0782070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0782243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0782263Z 
2022-11-23T02:25:23.0782371Z Running tests...
2022-11-23T02:25:23.0782637Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0782950Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0783264Z   test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0783516Z DDP works as expected when layer is checkpointed only once. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0783737Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42619
2022-11-23T02:25:23.0783955Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42620
2022-11-23T02:25:23.0784314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0784492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0784870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0785067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0785436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0785611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0785985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0786173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0786387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0786616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0786871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_r1gsj_0
2022-11-23T02:25:23.0787143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_r1gsj_0/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0787398Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphye1osl0
2022-11-23T02:25:23.0787672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphye1osl0/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0787910Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0788139Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0788371Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0788586Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0789573Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:25:23.0789699Z   warnings.warn(
2022-11-23T02:25:23.0790608Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:25:23.0790721Z   warnings.warn(
2022-11-23T02:25:23.0790953Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0791181Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0791414Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0791648Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0791930Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0792139Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0792364Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0792592Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0792693Z ok (6.342s)
2022-11-23T02:25:23.0792713Z 
2022-11-23T02:25:23.0792978Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0793091Z Ran 1 test in 6.343s
2022-11-23T02:25:23.0793111Z 
2022-11-23T02:25:23.0793201Z OK
2022-11-23T02:25:23.0793220Z 
2022-11-23T02:25:23.0793342Z Generating XML reports...
2022-11-23T02:25:23.0793809Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021108.xml
2022-11-23T02:25:23.0794169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0794347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0794729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0794923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0794943Z 
2022-11-23T02:25:23.0795050Z Running tests...
2022-11-23T02:25:23.0795312Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0795621Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0795887Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0796221Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0796445Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42839
2022-11-23T02:25:23.0796667Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42840
2022-11-23T02:25:23.0797043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0797220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0797600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0797791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0798155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0798409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0798773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0798966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0799196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0799426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0799684Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgby1_oi8
2022-11-23T02:25:23.0799954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgby1_oi8/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0800209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiaa7jekn
2022-11-23T02:25:23.0800477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiaa7jekn/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0800701Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0800985Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0801218Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0801454Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0801555Z ok (6.295s)
2022-11-23T02:25:23.0801575Z 
2022-11-23T02:25:23.0801843Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0801955Z Ran 1 test in 6.295s
2022-11-23T02:25:23.0801974Z 
2022-11-23T02:25:23.0802065Z OK
2022-11-23T02:25:23.0802084Z 
2022-11-23T02:25:23.0802188Z Generating XML reports...
2022-11-23T02:25:23.0802655Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021117.xml
2022-11-23T02:25:23.0803030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0803209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0803590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0803785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0803805Z 
2022-11-23T02:25:23.0803912Z Running tests...
2022-11-23T02:25:23.0804175Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0804488Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0804736Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0805089Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0805310Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43059
2022-11-23T02:25:23.0805531Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43060
2022-11-23T02:25:23.0805901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0806076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0806458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0806650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0807015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0807171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0807592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0807791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0808026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0808254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0808514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwxdq5bju
2022-11-23T02:25:23.0808787Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwxdq5bju/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0809040Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5z7f76ql
2022-11-23T02:25:23.0809287Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5z7f76ql/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0809524Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0809761Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0810047Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0810280Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0810382Z ok (6.307s)
2022-11-23T02:25:23.0810401Z 
2022-11-23T02:25:23.0810672Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0810784Z Ran 1 test in 6.307s
2022-11-23T02:25:23.0810803Z 
2022-11-23T02:25:23.0810875Z OK
2022-11-23T02:25:23.0810912Z 
2022-11-23T02:25:23.0811017Z Generating XML reports...
2022-11-23T02:25:23.0811482Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021126.xml
2022-11-23T02:25:23.0811855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0812034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0812414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0812610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0812629Z 
2022-11-23T02:25:23.0812737Z Running tests...
2022-11-23T02:25:23.0813001Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0813297Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0813545Z   test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0813919Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0814142Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43279
2022-11-23T02:25:23.0814364Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43280
2022-11-23T02:25:23.0814740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0814916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0815291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0815467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0815830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0816004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0816381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0816618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0817038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0817275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0817536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph5p2_ium
2022-11-23T02:25:23.0817806Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph5p2_ium/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0818043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4o736vhx
2022-11-23T02:25:23.0818308Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4o736vhx/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0818544Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0818783Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0819561Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:25:23.0820427Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:25:23.0820669Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0820905Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0821007Z ok (6.418s)
2022-11-23T02:25:23.0821026Z 
2022-11-23T02:25:23.0821305Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0821417Z Ran 1 test in 6.418s
2022-11-23T02:25:23.0821436Z 
2022-11-23T02:25:23.0821529Z OK
2022-11-23T02:25:23.0821548Z 
2022-11-23T02:25:23.0821652Z Generating XML reports...
2022-11-23T02:25:23.0822116Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021135.xml
2022-11-23T02:25:23.0822488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0822670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0823054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0823249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0823270Z 
2022-11-23T02:25:23.0823376Z Running tests...
2022-11-23T02:25:23.0823636Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0823928Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0824174Z   test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0824546Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0824767Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43499
2022-11-23T02:25:23.0825051Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43500
2022-11-23T02:25:23.0825437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0825617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0825998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0826191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0826544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0826721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0827094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0827288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0827521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0827801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0828061Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpif58ni_a
2022-11-23T02:25:23.0828335Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpif58ni_a/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0828576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphgveuv7x
2022-11-23T02:25:23.0828848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphgveuv7x/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0828951Z ok (6.315s)
2022-11-23T02:25:23.0828971Z 
2022-11-23T02:25:23.0829239Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0829349Z Ran 1 test in 6.315s
2022-11-23T02:25:23.0829368Z 
2022-11-23T02:25:23.0829464Z OK
2022-11-23T02:25:23.0829482Z 
2022-11-23T02:25:23.0829605Z Generating XML reports...
2022-11-23T02:25:23.0830071Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021144.xml
2022-11-23T02:25:23.0830440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0830598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0830974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0831167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0831186Z 
2022-11-23T02:25:23.0831294Z Running tests...
2022-11-23T02:25:23.0831558Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0831873Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0832117Z   test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0832392Z Checkpointing should work with static graph in the case of checkpointing ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0832594Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43719
2022-11-23T02:25:23.0832812Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43720
2022-11-23T02:25:23.0833185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0833363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0833742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0833934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0834349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0834533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0834906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0835079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0835311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0835541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0835800Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2u1q5z4w
2022-11-23T02:25:23.0836072Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2u1q5z4w/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0836333Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptt92cy67
2022-11-23T02:25:23.0836599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptt92cy67/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0836887Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0837109Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0837343Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0837578Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0837681Z ok (6.178s)
2022-11-23T02:25:23.0837700Z 
2022-11-23T02:25:23.0837970Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0838081Z Ran 1 test in 6.178s
2022-11-23T02:25:23.0838100Z 
2022-11-23T02:25:23.0838191Z OK
2022-11-23T02:25:23.0838211Z 
2022-11-23T02:25:23.0838334Z Generating XML reports...
2022-11-23T02:25:23.0838804Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021153.xml
2022-11-23T02:25:23.0839162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0839340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0839721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0839922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0839941Z 
2022-11-23T02:25:23.0840049Z Running tests...
2022-11-23T02:25:23.0840313Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0840623Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0841012Z   test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0841265Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0841490Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43939
2022-11-23T02:25:23.0841709Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43940
2022-11-23T02:25:23.0842082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0842257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0842636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0842827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0843189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0843408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0843779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0843969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0844199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0844700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0844967Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps37cr842
2022-11-23T02:25:23.0845224Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu_j2xa5s
2022-11-23T02:25:23.0845493Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps37cr842/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0845760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu_j2xa5s/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0846543Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:25:23.0847384Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:25:23.0848312Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:25:23.0848427Z   warnings.warn(
2022-11-23T02:25:23.0849341Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:25:23.0849455Z   warnings.warn(
2022-11-23T02:25:23.0849680Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0849923Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0850159Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0850396Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0850497Z ok (6.401s)
2022-11-23T02:25:23.0850517Z 
2022-11-23T02:25:23.0850787Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0850898Z Ran 1 test in 6.401s
2022-11-23T02:25:23.0850917Z 
2022-11-23T02:25:23.0851010Z OK
2022-11-23T02:25:23.0851029Z 
2022-11-23T02:25:23.0851135Z Generating XML reports...
2022-11-23T02:25:23.0851645Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021201.xml
2022-11-23T02:25:23.0852029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0852211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0852590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0852782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0852801Z 
2022-11-23T02:25:23.0852909Z Running tests...
2022-11-23T02:25:23.0853173Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0853484Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0853724Z   test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0854000Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0854219Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44159
2022-11-23T02:25:23.0854488Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44160
2022-11-23T02:25:23.0854858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0855035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0855414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0855604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0855946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0856121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0856494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0856861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0857106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0857337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0857595Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpclba84eq
2022-11-23T02:25:23.0857864Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpclba84eq/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0858115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_oyoggzk
2022-11-23T02:25:23.0858366Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_oyoggzk/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0859285Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:25:23.0859404Z   warnings.warn(
2022-11-23T02:25:23.0860309Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:25:23.0860422Z   warnings.warn(
2022-11-23T02:25:23.0860657Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0860961Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0861210Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0861444Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0861546Z ok (6.291s)
2022-11-23T02:25:23.0861565Z 
2022-11-23T02:25:23.0861893Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0862014Z Ran 1 test in 6.291s
2022-11-23T02:25:23.0862034Z 
2022-11-23T02:25:23.0862127Z OK
2022-11-23T02:25:23.0862145Z 
2022-11-23T02:25:23.0862269Z Generating XML reports...
2022-11-23T02:25:23.0862733Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021210.xml
2022-11-23T02:25:23.0863103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0863284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0863667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0863929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0863949Z 
2022-11-23T02:25:23.0864039Z Running tests...
2022-11-23T02:25:23.0864306Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0864617Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0864877Z   test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0865115Z Test that checkpointing with weight sharing works. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0865335Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44379
2022-11-23T02:25:23.0865557Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44380
2022-11-23T02:25:23.0865931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0866092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0866472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0866663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0867027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0867200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0867575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0867763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0867996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0868203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0868461Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcbbc59kn
2022-11-23T02:25:23.0868731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcbbc59kn/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0868987Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmpxwif4g
2022-11-23T02:25:23.0869259Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmpxwif4g/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0869494Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0869731Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0870014Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0870258Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0870345Z ok (6.331s)
2022-11-23T02:25:23.0870365Z 
2022-11-23T02:25:23.0870633Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0870744Z Ran 1 test in 6.332s
2022-11-23T02:25:23.0870763Z 
2022-11-23T02:25:23.0870854Z OK
2022-11-23T02:25:23.0870873Z 
2022-11-23T02:25:23.0870997Z Generating XML reports...
2022-11-23T02:25:23.0871457Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021219.xml
2022-11-23T02:25:23.0871826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0872000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0872366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0872561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0872648Z 
2022-11-23T02:25:23.0872760Z Running tests...
2022-11-23T02:25:23.0873024Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0873335Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0873594Z   test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0873831Z Test that checkpointing with weight sharing works. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0874052Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44599
2022-11-23T02:25:23.0874272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44600
2022-11-23T02:25:23.0874631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0874808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0875189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0875385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0875751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0875924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0876294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0876484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0876698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0876929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0877188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6gwfm4a8
2022-11-23T02:25:23.0877459Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6gwfm4a8/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0877714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq0d98xxn
2022-11-23T02:25:23.0877983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq0d98xxn/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0878216Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0878451Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0878685Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0878950Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0879186Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0879419Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0879650Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0879878Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0879980Z ok (6.304s)
2022-11-23T02:25:23.0879999Z 
2022-11-23T02:25:23.0880266Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0880377Z Ran 1 test in 6.304s
2022-11-23T02:25:23.0880396Z 
2022-11-23T02:25:23.0880469Z OK
2022-11-23T02:25:23.0880488Z 
2022-11-23T02:25:23.0880612Z Generating XML reports...
2022-11-23T02:25:23.0881082Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021228.xml
2022-11-23T02:25:23.0881458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0881701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0882080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0882274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0882293Z 
2022-11-23T02:25:23.0882401Z Running tests...
2022-11-23T02:25:23.0882660Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0882954Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0883245Z   test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0883467Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44819
2022-11-23T02:25:23.0883688Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44820
2022-11-23T02:25:23.0884064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0884240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0884621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0884812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0885157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0885331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0885704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0885896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0886131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0886365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0886623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplw_mmfoz
2022-11-23T02:25:23.0886891Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplw_mmfoz/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0887147Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp78bac3co
2022-11-23T02:25:23.0887399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp78bac3co/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0887500Z ok (6.912s)
2022-11-23T02:25:23.0887520Z 
2022-11-23T02:25:23.0887790Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0887949Z Ran 1 test in 6.912s
2022-11-23T02:25:23.0887971Z 
2022-11-23T02:25:23.0888067Z OK
2022-11-23T02:25:23.0888089Z 
2022-11-23T02:25:23.0888215Z Generating XML reports...
2022-11-23T02:25:23.0888677Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021236.xml
2022-11-23T02:25:23.0889061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0889220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0889602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0889793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0889814Z 
2022-11-23T02:25:23.0889921Z Running tests...
2022-11-23T02:25:23.0890186Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0890501Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0890856Z   test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0891078Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45040
2022-11-23T02:25:23.0891298Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45041
2022-11-23T02:25:23.0891653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0891827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0892204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0892397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0892765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0892943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0893313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0893501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0893714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0893948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0894205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnw461s0k
2022-11-23T02:25:23.0894473Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnw461s0k/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0894736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqavsugg2
2022-11-23T02:25:23.0895008Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqavsugg2/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0895112Z ok (6.878s)
2022-11-23T02:25:23.0895132Z 
2022-11-23T02:25:23.0895398Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0895508Z Ran 1 test in 6.879s
2022-11-23T02:25:23.0895527Z 
2022-11-23T02:25:23.0895600Z OK
2022-11-23T02:25:23.0895619Z 
2022-11-23T02:25:23.0895742Z Generating XML reports...
2022-11-23T02:25:23.0896205Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021246.xml
2022-11-23T02:25:23.0896577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0896928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0897399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0897600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0897625Z 
2022-11-23T02:25:23.0897732Z Running tests...
2022-11-23T02:25:23.0897981Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0898294Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0898601Z   test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0898821Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45261
2022-11-23T02:25:23.0899039Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45262
2022-11-23T02:25:23.0899405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0899585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0899966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0900224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0900575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0900748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0901118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0901306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0901538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0901763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0902029Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx9wqirk5
2022-11-23T02:25:23.0902306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx9wqirk5/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0902542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_vin_e4r
2022-11-23T02:25:23.0902808Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_vin_e4r/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0902911Z ok (6.881s)
2022-11-23T02:25:23.0902930Z 
2022-11-23T02:25:23.0903197Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0903309Z Ran 1 test in 6.881s
2022-11-23T02:25:23.0903328Z 
2022-11-23T02:25:23.0903419Z OK
2022-11-23T02:25:23.0903438Z 
2022-11-23T02:25:23.0903561Z Generating XML reports...
2022-11-23T02:25:23.0904022Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021255.xml
2022-11-23T02:25:23.0904396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0904557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0904935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0905126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0905146Z 
2022-11-23T02:25:23.0905256Z Running tests...
2022-11-23T02:25:23.0905526Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0905841Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0906077Z   test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0906410Z This unit test verifies whether a DDP communication hook that calls allreduce and then ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0906618Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45482
2022-11-23T02:25:23.0906840Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45483
2022-11-23T02:25:23.0907212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0907392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0907770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0907963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0908325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0908497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0908879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0909102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0909335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0909568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0909826Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdsddk_nr
2022-11-23T02:25:23.0910097Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdsddk_nr/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0910351Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2hbndmz0
2022-11-23T02:25:23.0910620Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2hbndmz0/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0910721Z ok (6.899s)
2022-11-23T02:25:23.0910741Z 
2022-11-23T02:25:23.0910998Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0911113Z Ran 1 test in 6.899s
2022-11-23T02:25:23.0911131Z 
2022-11-23T02:25:23.0911225Z OK
2022-11-23T02:25:23.0911244Z 
2022-11-23T02:25:23.0911368Z Generating XML reports...
2022-11-23T02:25:23.0911835Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021305.xml
2022-11-23T02:25:23.0912207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0912381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0912756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0912946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0912965Z 
2022-11-23T02:25:23.0913055Z Running tests...
2022-11-23T02:25:23.0913321Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0913637Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0913869Z   test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0914159Z This unit test verifies whether the Future object is passed properly using nccl backend. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0914381Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45703
2022-11-23T02:25:23.0914599Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45704
2022-11-23T02:25:23.0914970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0915128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0915556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0915753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0916124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0916298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0916669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0916858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0917090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0917308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0917569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg9vyt5j2
2022-11-23T02:25:23.0917857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg9vyt5j2/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0918161Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpenhfqzb_
2022-11-23T02:25:23.0918429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpenhfqzb_/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0918530Z ok (7.007s)
2022-11-23T02:25:23.0918549Z 
2022-11-23T02:25:23.0918814Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0918923Z Ran 1 test in 7.007s
2022-11-23T02:25:23.0918942Z 
2022-11-23T02:25:23.0919014Z OK
2022-11-23T02:25:23.0919033Z 
2022-11-23T02:25:23.0919153Z Generating XML reports...
2022-11-23T02:25:23.0919614Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021314.xml
2022-11-23T02:25:23.0919984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0920158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0920537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0920729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0920749Z 
2022-11-23T02:25:23.0920853Z Running tests...
2022-11-23T02:25:23.0921100Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0921406Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0921690Z   test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0921907Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45924
2022-11-23T02:25:23.0922121Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45925
2022-11-23T02:25:23.0922493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0922670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0923049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0923236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0923586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0923756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0924129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0924317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0924593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0924826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0924929Z ok (6.932s)
2022-11-23T02:25:23.0924949Z 
2022-11-23T02:25:23.0925215Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0925307Z Ran 1 test in 6.933s
2022-11-23T02:25:23.0925343Z 
2022-11-23T02:25:23.0925417Z OK
2022-11-23T02:25:23.0925435Z 
2022-11-23T02:25:23.0925556Z Generating XML reports...
2022-11-23T02:25:23.0926013Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021324.xml
2022-11-23T02:25:23.0926379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0926551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0926927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0927168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0927188Z 
2022-11-23T02:25:23.0927296Z Running tests...
2022-11-23T02:25:23.0927543Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0927853Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0928058Z   test_ddp_packed_sequence (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.0928315Z Tests that DDP with ``device_ids`` specified can run a forward and ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0928535Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46131
2022-11-23T02:25:23.0928752Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46132
2022-11-23T02:25:23.0929126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0929303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0929668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0929858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0930218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0930389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0930760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0930946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0931175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0931422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.0931652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0931876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.0932279Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0932676Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.0932930Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps_dbff7w
2022-11-23T02:25:23.0933199Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps_dbff7w/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0933450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7vs73awq
2022-11-23T02:25:23.0933764Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7vs73awq/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0934724Z /opt/conda/lib/python3.10/site-packages/torch/distributed/_shard/replicated_tensor.py:113: UserWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greatly increasing memory usage. To compact weights again call flatten_parameters(). (Triggered internally at /var/lib/jenkins/workspace/aten/src/ATen/native/cudnn/RNN.cpp:982.)
2022-11-23T02:25:23.0934843Z   rs = func(*args, **kwargs)
2022-11-23T02:25:23.0935784Z /opt/conda/lib/python3.10/site-packages/torch/distributed/_shard/replicated_tensor.py:113: UserWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greatly increasing memory usage. To compact weights again call flatten_parameters(). (Triggered internally at /var/lib/jenkins/workspace/aten/src/ATen/native/cudnn/RNN.cpp:982.)
2022-11-23T02:25:23.0935904Z   rs = func(*args, **kwargs)
2022-11-23T02:25:23.0936081Z ok (7.813s)
2022-11-23T02:25:23.0936102Z 
2022-11-23T02:25:23.0936369Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0936479Z Ran 1 test in 7.813s
2022-11-23T02:25:23.0936498Z 
2022-11-23T02:25:23.0936588Z OK
2022-11-23T02:25:23.0936606Z 
2022-11-23T02:25:23.0936904Z Generating XML reports...
2022-11-23T02:25:23.0937382Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021333.xml
2022-11-23T02:25:23.0937750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0937926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0938290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0938491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0938514Z 
2022-11-23T02:25:23.0938621Z Running tests...
2022-11-23T02:25:23.0938885Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0939193Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0939462Z   test_ddp_weight_sharing (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0939680Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46352
2022-11-23T02:25:23.0939896Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46353
2022-11-23T02:25:23.0940267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0940424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0940805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0940997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0941358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0941528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0941896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0942081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0942307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0942512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0942840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx_zihahl
2022-11-23T02:25:23.0943121Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx_zihahl/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0943377Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpolen0kyb
2022-11-23T02:25:23.0943647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpolen0kyb/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0943883Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0944119Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0944349Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0944576Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0944792Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0945027Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0945327Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0945550Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0945650Z ok (7.170s)
2022-11-23T02:25:23.0945670Z 
2022-11-23T02:25:23.0945939Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0946050Z Ran 1 test in 7.170s
2022-11-23T02:25:23.0946069Z 
2022-11-23T02:25:23.0946159Z OK
2022-11-23T02:25:23.0946178Z 
2022-11-23T02:25:23.0946282Z Generating XML reports...
2022-11-23T02:25:23.0946745Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021343.xml
2022-11-23T02:25:23.0947115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0947294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0947671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0947866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0947885Z 
2022-11-23T02:25:23.0947993Z Running tests...
2022-11-23T02:25:23.0948255Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0948548Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0948822Z   test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0949043Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46573
2022-11-23T02:25:23.0949261Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46574
2022-11-23T02:25:23.0949635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0949812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0950189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0950380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0950742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0950897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0951274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0951460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0951737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0952284Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/lazy.py:180: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment.
2022-11-23T02:25:23.0952564Z   warnings.warn('Lazy modules are a new feature under heavy development '
2022-11-23T02:25:23.0952794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0953314Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/lazy.py:180: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment.
2022-11-23T02:25:23.0953585Z   warnings.warn('Lazy modules are a new feature under heavy development '
2022-11-23T02:25:23.0953828Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzrpchpy9
2022-11-23T02:25:23.0954104Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzrpchpy9/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0954357Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd9j4pyxq
2022-11-23T02:25:23.0954676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd9j4pyxq/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0954778Z ok (4.084s)
2022-11-23T02:25:23.0954798Z 
2022-11-23T02:25:23.0955062Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0955172Z Ran 1 test in 4.084s
2022-11-23T02:25:23.0955192Z 
2022-11-23T02:25:23.0955283Z OK
2022-11-23T02:25:23.0955301Z 
2022-11-23T02:25:23.0955423Z Generating XML reports...
2022-11-23T02:25:23.0955865Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021353.xml
2022-11-23T02:25:23.0956234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0956412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0956792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0956986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0957005Z 
2022-11-23T02:25:23.0957111Z Running tests...
2022-11-23T02:25:23.0957373Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0957683Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0957947Z   test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0958166Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46776
2022-11-23T02:25:23.0958381Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46777
2022-11-23T02:25:23.0958754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0958930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0959317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0959515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0959878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0960054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0960417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0960608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0960900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0961178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0961444Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphfthqgyo
2022-11-23T02:25:23.0961716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphfthqgyo/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0962060Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6od_hm1f
2022-11-23T02:25:23.0962334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6od_hm1f/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0962417Z ok (6.964s)
2022-11-23T02:25:23.0962454Z 
2022-11-23T02:25:23.0962712Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0962821Z Ran 1 test in 6.964s
2022-11-23T02:25:23.0962840Z 
2022-11-23T02:25:23.0962933Z OK
2022-11-23T02:25:23.0962953Z 
2022-11-23T02:25:23.0963076Z Generating XML reports...
2022-11-23T02:25:23.0963549Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021400.xml
2022-11-23T02:25:23.0963977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0964151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0964525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0964700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0964719Z 
2022-11-23T02:25:23.0964825Z Running tests...
2022-11-23T02:25:23.0965084Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0965390Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0965679Z   test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0965898Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46997
2022-11-23T02:25:23.0966117Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46998
2022-11-23T02:25:23.0966485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0966642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0967019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0967208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0967575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0967750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0968125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0968318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0968547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0968773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0969014Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg0hp1vxb
2022-11-23T02:25:23.0969286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg0hp1vxb/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0969540Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf11mn6va
2022-11-23T02:25:23.0969806Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf11mn6va/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0969906Z ok (6.844s)
2022-11-23T02:25:23.0969927Z 
2022-11-23T02:25:23.0970236Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0970356Z Ran 1 test in 6.844s
2022-11-23T02:25:23.0970375Z 
2022-11-23T02:25:23.0970466Z OK
2022-11-23T02:25:23.0970485Z 
2022-11-23T02:25:23.0970589Z Generating XML reports...
2022-11-23T02:25:23.0971053Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021409.xml
2022-11-23T02:25:23.0971421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0971595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0971968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0972158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0972177Z 
2022-11-23T02:25:23.0972285Z Running tests...
2022-11-23T02:25:23.0972551Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0972913Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0973164Z   test_failure_recovery (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0973382Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47218
2022-11-23T02:25:23.0973600Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47219
2022-11-23T02:25:23.0973968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0974143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0974520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0974714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0975079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0975239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0975612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0975798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0976028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.0976255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.0976581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph_oagqxy
2022-11-23T02:25:23.0977027Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph_oagqxy/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0977291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqhz6krtn
2022-11-23T02:25:23.0977563Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqhz6krtn/_remote_module_non_scriptable.py
2022-11-23T02:25:23.0977782Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0978018Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0978250Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0978482Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.0978584Z ok (7.515s)
2022-11-23T02:25:23.0978604Z 
2022-11-23T02:25:23.0978879Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0978989Z Ran 1 test in 7.515s
2022-11-23T02:25:23.0979008Z 
2022-11-23T02:25:23.0979100Z OK
2022-11-23T02:25:23.0979119Z 
2022-11-23T02:25:23.0979298Z Generating XML reports...
2022-11-23T02:25:23.0979775Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021418.xml
2022-11-23T02:25:23.0980146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0980321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0980698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0980887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0980906Z 
2022-11-23T02:25:23.0981012Z Running tests...
2022-11-23T02:25:23.0981279Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0981593Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0981882Z   test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0982716Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82632 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.691s)
2022-11-23T02:25:23.0982739Z 
2022-11-23T02:25:23.0983004Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0983116Z Ran 1 test in 1.691s
2022-11-23T02:25:23.0983135Z 
2022-11-23T02:25:23.0983241Z OK (skipped=1)
2022-11-23T02:25:23.0983259Z 
2022-11-23T02:25:23.0983382Z Generating XML reports...
2022-11-23T02:25:23.0983836Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021428.xml
2022-11-23T02:25:23.0984208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0984386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0984766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0984939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0984959Z 
2022-11-23T02:25:23.0985065Z Running tests...
2022-11-23T02:25:23.0985325Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0985632Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0985933Z   test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0986678Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/83301 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.701s)
2022-11-23T02:25:23.0986701Z 
2022-11-23T02:25:23.0986964Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0987081Z Ran 1 test in 1.701s
2022-11-23T02:25:23.0987100Z 
2022-11-23T02:25:23.0987204Z OK (skipped=1)
2022-11-23T02:25:23.0987224Z 
2022-11-23T02:25:23.0987329Z Generating XML reports...
2022-11-23T02:25:23.0987785Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021433.xml
2022-11-23T02:25:23.0988153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0988325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0988751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0988948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0988971Z 
2022-11-23T02:25:23.0989078Z Running tests...
2022-11-23T02:25:23.0989340Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0989649Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0989927Z   test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0990668Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82385 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.688s)
2022-11-23T02:25:23.0990688Z 
2022-11-23T02:25:23.0990952Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0991063Z Ran 1 test in 1.688s
2022-11-23T02:25:23.0991083Z 
2022-11-23T02:25:23.0991238Z OK (skipped=1)
2022-11-23T02:25:23.0991257Z 
2022-11-23T02:25:23.0991384Z Generating XML reports...
2022-11-23T02:25:23.0991841Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021437.xml
2022-11-23T02:25:23.0992210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0992386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0992767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0992942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0992961Z 
2022-11-23T02:25:23.0993069Z Running tests...
2022-11-23T02:25:23.0993333Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0993644Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0993960Z   test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0994697Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82979 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.700s)
2022-11-23T02:25:23.0994717Z 
2022-11-23T02:25:23.0994980Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0995090Z Ran 1 test in 1.700s
2022-11-23T02:25:23.0995109Z 
2022-11-23T02:25:23.0995214Z OK (skipped=1)
2022-11-23T02:25:23.0995233Z 
2022-11-23T02:25:23.0995338Z Generating XML reports...
2022-11-23T02:25:23.0995801Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021441.xml
2022-11-23T02:25:23.0996174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.0996347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.0996724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.0996917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.0996937Z 
2022-11-23T02:25:23.0997044Z Running tests...
2022-11-23T02:25:23.0997306Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0997617Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.0997965Z   test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.0998714Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82400 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.653s)
2022-11-23T02:25:23.0998738Z 
2022-11-23T02:25:23.0998998Z ----------------------------------------------------------------------
2022-11-23T02:25:23.0999109Z Ran 1 test in 1.653s
2022-11-23T02:25:23.0999129Z 
2022-11-23T02:25:23.0999231Z OK (skipped=1)
2022-11-23T02:25:23.0999250Z 
2022-11-23T02:25:23.0999371Z Generating XML reports...
2022-11-23T02:25:23.0999825Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021445.xml
2022-11-23T02:25:23.1000198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1000374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1000806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1000980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1000999Z 
2022-11-23T02:25:23.1001105Z Running tests...
2022-11-23T02:25:23.1001365Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1001672Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1001980Z   test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1002723Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82500 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.718s)
2022-11-23T02:25:23.1002746Z 
2022-11-23T02:25:23.1003007Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1003117Z Ran 1 test in 1.718s
2022-11-23T02:25:23.1003136Z 
2022-11-23T02:25:23.1003242Z OK (skipped=1)
2022-11-23T02:25:23.1003261Z 
2022-11-23T02:25:23.1003384Z Generating XML reports...
2022-11-23T02:25:23.1003825Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021449.xml
2022-11-23T02:25:23.1004192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1004364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1004739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1004931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1004953Z 
2022-11-23T02:25:23.1005058Z Running tests...
2022-11-23T02:25:23.1005316Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1005623Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1005852Z   test_fp16 (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1006072Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47846
2022-11-23T02:25:23.1006291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47847
2022-11-23T02:25:23.1006657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1006833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1007259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1007459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1007823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1007993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1008350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1008539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1008765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1008994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1009254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps49rs6z1
2022-11-23T02:25:23.1009523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps49rs6z1/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1009840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp71kiejsy
2022-11-23T02:25:23.1010108Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp71kiejsy/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1010191Z ok (7.438s)
2022-11-23T02:25:23.1010211Z 
2022-11-23T02:25:23.1010476Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1010584Z Ran 1 test in 7.438s
2022-11-23T02:25:23.1010603Z 
2022-11-23T02:25:23.1010692Z OK
2022-11-23T02:25:23.1010711Z 
2022-11-23T02:25:23.1010832Z Generating XML reports...
2022-11-23T02:25:23.1011290Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021453.xml
2022-11-23T02:25:23.1011660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1011833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1012214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1012390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1012410Z 
2022-11-23T02:25:23.1012515Z Running tests...
2022-11-23T02:25:23.1012774Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1013080Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1013364Z   test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1013583Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48067
2022-11-23T02:25:23.1013805Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48068
2022-11-23T02:25:23.1014178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1014339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1014718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1014906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1015270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1015440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1015810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1015998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1016282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1017009Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.1017243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1017759Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.1018069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpopapew3f
2022-11-23T02:25:23.1018350Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpopapew3f/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1018684Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpymltv3rc
2022-11-23T02:25:23.1018953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpymltv3rc/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1019055Z ok (6.921s)
2022-11-23T02:25:23.1019076Z 
2022-11-23T02:25:23.1019351Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1019462Z Ran 1 test in 6.921s
2022-11-23T02:25:23.1019481Z 
2022-11-23T02:25:23.1019572Z OK
2022-11-23T02:25:23.1019591Z 
2022-11-23T02:25:23.1019695Z Generating XML reports...
2022-11-23T02:25:23.1020159Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021503.xml
2022-11-23T02:25:23.1020534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1020710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1021089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1021279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1021299Z 
2022-11-23T02:25:23.1021406Z Running tests...
2022-11-23T02:25:23.1021667Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1021978Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1022240Z   test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1022459Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48288
2022-11-23T02:25:23.1022678Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48289
2022-11-23T02:25:23.1023049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1023224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1023599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1023786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1024149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1024305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1024678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1024866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1025156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1025709Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.1025934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1026476Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.1026736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpae5a9dd0
2022-11-23T02:25:23.1027051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpae5a9dd0/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1027306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmo4uudht
2022-11-23T02:25:23.1027574Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmo4uudht/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1027657Z ok (6.914s)
2022-11-23T02:25:23.1027677Z 
2022-11-23T02:25:23.1027947Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1028056Z Ran 1 test in 6.915s
2022-11-23T02:25:23.1028075Z 
2022-11-23T02:25:23.1028165Z OK
2022-11-23T02:25:23.1028183Z 
2022-11-23T02:25:23.1028307Z Generating XML reports...
2022-11-23T02:25:23.1028767Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021513.xml
2022-11-23T02:25:23.1029141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1029317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1029679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1029872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1029892Z 
2022-11-23T02:25:23.1029996Z Running tests...
2022-11-23T02:25:23.1030258Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1030565Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1030829Z   test_fp16_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1031049Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48509
2022-11-23T02:25:23.1031268Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48510
2022-11-23T02:25:23.1031643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1031801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1032177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1032366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1032728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1032899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1033270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1033505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1033741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1033954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1034212Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9pxjqkt7
2022-11-23T02:25:23.1034482Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9pxjqkt7/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1034738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmidbns8k
2022-11-23T02:25:23.1035006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmidbns8k/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1035106Z ok (7.390s)
2022-11-23T02:25:23.1035126Z 
2022-11-23T02:25:23.1035399Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1035511Z Ran 1 test in 7.390s
2022-11-23T02:25:23.1035530Z 
2022-11-23T02:25:23.1035607Z OK
2022-11-23T02:25:23.1035643Z 
2022-11-23T02:25:23.1035748Z Generating XML reports...
2022-11-23T02:25:23.1036262Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021522.xml
2022-11-23T02:25:23.1036634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1036809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1037184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1037373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1037393Z 
2022-11-23T02:25:23.1037499Z Running tests...
2022-11-23T02:25:23.1037759Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1038055Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1038368Z   test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1038593Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48730
2022-11-23T02:25:23.1038811Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48731
2022-11-23T02:25:23.1039180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1039353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1039729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1039917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1040285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1040440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1040815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1041007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1041234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1041460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1041748Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd4gytyni
2022-11-23T02:25:23.1042016Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd4gytyni/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1042268Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpknee5gkv
2022-11-23T02:25:23.1042571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpknee5gkv/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1042820Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1043056Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1043289Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1043520Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1043748Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1043978Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1044203Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1044433Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1044646Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1044927Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1045155Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1045381Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1045610Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1045833Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1046055Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1046282Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1046494Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1046721Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1046946Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1047167Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1047394Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1047626Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1047858Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1048080Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1048285Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1048517Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1048743Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1048972Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1049193Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1049411Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1049635Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1049865Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1050095Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1050300Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1050588Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1050828Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1050932Z ok (9.126s)
2022-11-23T02:25:23.1050952Z 
2022-11-23T02:25:23.1051229Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1051344Z Ran 1 test in 9.127s
2022-11-23T02:25:23.1051364Z 
2022-11-23T02:25:23.1051457Z OK
2022-11-23T02:25:23.1051476Z 
2022-11-23T02:25:23.1051603Z Generating XML reports...
2022-11-23T02:25:23.1052048Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021532.xml
2022-11-23T02:25:23.1052420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1052600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1052988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1053231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1053251Z 
2022-11-23T02:25:23.1053364Z Running tests...
2022-11-23T02:25:23.1053631Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1053950Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1054238Z   test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1054440Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48951
2022-11-23T02:25:23.1054662Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48952
2022-11-23T02:25:23.1055033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1055214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1055585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1055770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1056148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1056342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1056929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1057139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1057370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1057610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1057870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxhsril5n
2022-11-23T02:25:23.1058155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxhsril5n/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1058413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgllzpiuf
2022-11-23T02:25:23.1058691Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgllzpiuf/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1059056Z [W logger.cpp:318] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-11-23T02:25:23.1059388Z [W logger.cpp:318] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-11-23T02:25:23.1059632Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1059941Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1060186Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1060427Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1060658Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1060887Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1061117Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1061327Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1061558Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1061781Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1062097Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1062336Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1062635Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1062860Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1063092Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1063301Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1063532Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1063763Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1063993Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1064226Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1064450Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1064682Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1064917Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1065146Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1065353Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1065583Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1065813Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1066044Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1066278Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1066505Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1066729Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1066956Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1067164Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1067389Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1067612Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1067842Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1067950Z ok (11.430s)
2022-11-23T02:25:23.1067971Z 
2022-11-23T02:25:23.1068300Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1068425Z Ran 1 test in 11.431s
2022-11-23T02:25:23.1068448Z 
2022-11-23T02:25:23.1068546Z OK
2022-11-23T02:25:23.1068565Z 
2022-11-23T02:25:23.1068692Z Generating XML reports...
2022-11-23T02:25:23.1069142Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021544.xml
2022-11-23T02:25:23.1069512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1069690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1070072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1070270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1070289Z 
2022-11-23T02:25:23.1070401Z Running tests...
2022-11-23T02:25:23.1070674Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1070993Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1071321Z   test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1071541Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49181
2022-11-23T02:25:23.1071759Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49182
2022-11-23T02:25:23.1072129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1072309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1072689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1072882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1073250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1073430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1073792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1073979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1074206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1074750Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.1075296Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.1075830Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.1076445Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.1076985Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.1077525Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.1077755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1078288Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.1078870Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.1079396Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.1079937Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.1080469Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.1081010Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.1081111Z ok (4.072s)
2022-11-23T02:25:23.1081132Z 
2022-11-23T02:25:23.1081407Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1081518Z Ran 1 test in 4.072s
2022-11-23T02:25:23.1081537Z 
2022-11-23T02:25:23.1081629Z OK
2022-11-23T02:25:23.1081648Z 
2022-11-23T02:25:23.1081753Z Generating XML reports...
2022-11-23T02:25:23.1082212Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021558.xml
2022-11-23T02:25:23.1082581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1082754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1083135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1083373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1083395Z 
2022-11-23T02:25:23.1083505Z Running tests...
2022-11-23T02:25:23.1083774Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1084082Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1084356Z   test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1084576Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49380
2022-11-23T02:25:23.1084793Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49381
2022-11-23T02:25:23.1085162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1085335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1085716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1085905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1086323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1086478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1086850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1087037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1087267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1087488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1087743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc9w2vbt4
2022-11-23T02:25:23.1088018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc9w2vbt4/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1088276Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmo0lrss6
2022-11-23T02:25:23.1088543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmo0lrss6/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1088627Z ok (7.415s)
2022-11-23T02:25:23.1088647Z 
2022-11-23T02:25:23.1088913Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1089022Z Ran 1 test in 7.415s
2022-11-23T02:25:23.1089041Z 
2022-11-23T02:25:23.1089131Z OK
2022-11-23T02:25:23.1089150Z 
2022-11-23T02:25:23.1089272Z Generating XML reports...
2022-11-23T02:25:23.1089733Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021604.xml
2022-11-23T02:25:23.1090105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1090284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1090650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1090839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1090859Z 
2022-11-23T02:25:23.1090963Z Running tests...
2022-11-23T02:25:23.1091221Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1091531Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1091841Z   test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1092062Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49601
2022-11-23T02:25:23.1092281Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49602
2022-11-23T02:25:23.1092697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1092864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1093242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1093433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1093793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1093966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1094334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1094525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1094764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1094977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1095293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptj5acp4d
2022-11-23T02:25:23.1095569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptj5acp4d/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1095826Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6bj6rtqw
2022-11-23T02:25:23.1096097Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6bj6rtqw/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1096201Z ok (7.430s)
2022-11-23T02:25:23.1096221Z 
2022-11-23T02:25:23.1096490Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1096604Z Ran 1 test in 7.430s
2022-11-23T02:25:23.1096622Z 
2022-11-23T02:25:23.1096898Z OK
2022-11-23T02:25:23.1096920Z 
2022-11-23T02:25:23.1097036Z Generating XML reports...
2022-11-23T02:25:23.1097518Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021614.xml
2022-11-23T02:25:23.1097899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1098073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1098448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1098638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1098658Z 
2022-11-23T02:25:23.1098766Z Running tests...
2022-11-23T02:25:23.1099026Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1099317Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1099627Z   test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1099849Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49822
2022-11-23T02:25:23.1100065Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49823
2022-11-23T02:25:23.1100434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1100608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1100989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1101190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1101562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1101795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1102190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1102386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1102620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1102857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1103118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_odxdblh
2022-11-23T02:25:23.1103395Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_odxdblh/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1103649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ooasitz
2022-11-23T02:25:23.1103922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ooasitz/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1104140Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1104448Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1104552Z ok (7.437s)
2022-11-23T02:25:23.1104572Z 
2022-11-23T02:25:23.1104842Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1104958Z Ran 1 test in 7.437s
2022-11-23T02:25:23.1104977Z 
2022-11-23T02:25:23.1105074Z OK
2022-11-23T02:25:23.1105093Z 
2022-11-23T02:25:23.1105220Z Generating XML reports...
2022-11-23T02:25:23.1105685Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021624.xml
2022-11-23T02:25:23.1106039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1106217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1106605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1106804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1106824Z 
2022-11-23T02:25:23.1106937Z Running tests...
2022-11-23T02:25:23.1107210Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1107524Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1107840Z   test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1108066Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50043
2022-11-23T02:25:23.1108269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50044
2022-11-23T02:25:23.1108645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1108818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1109197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1109385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1109743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1109914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1110285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1110455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1110685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1110956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1111223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxymsbgl8
2022-11-23T02:25:23.1111495Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxymsbgl8/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1111749Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk3o15wn4
2022-11-23T02:25:23.1112015Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk3o15wn4/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1112253Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1112490Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1112573Z ok (7.439s)
2022-11-23T02:25:23.1112593Z 
2022-11-23T02:25:23.1112862Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1112978Z Ran 1 test in 7.439s
2022-11-23T02:25:23.1112998Z 
2022-11-23T02:25:23.1113087Z OK
2022-11-23T02:25:23.1113149Z 
2022-11-23T02:25:23.1113275Z Generating XML reports...
2022-11-23T02:25:23.1113743Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021634.xml
2022-11-23T02:25:23.1114117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1114292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1114653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1114842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1114862Z 
2022-11-23T02:25:23.1114967Z Running tests...
2022-11-23T02:25:23.1115227Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1115541Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1115819Z   test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1116035Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50264
2022-11-23T02:25:23.1116252Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50265
2022-11-23T02:25:23.1116620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1116779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1117155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1117345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1117711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1117885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1118261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1118448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1118677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1118889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1119144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqr758nkg
2022-11-23T02:25:23.1119414Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqr758nkg/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1119665Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphve4jf0q
2022-11-23T02:25:23.1136655Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphve4jf0q/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1137499Z [W logger.cpp:318] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-11-23T02:25:23.1137877Z [W logger.cpp:318] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-11-23T02:25:23.1138133Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1138382Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1138486Z ok (8.977s)
2022-11-23T02:25:23.1138508Z 
2022-11-23T02:25:23.1138777Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1138892Z Ran 1 test in 8.977s
2022-11-23T02:25:23.1138913Z 
2022-11-23T02:25:23.1139003Z OK
2022-11-23T02:25:23.1139023Z 
2022-11-23T02:25:23.1139150Z Generating XML reports...
2022-11-23T02:25:23.1139647Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021644.xml
2022-11-23T02:25:23.1140169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1140353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1140761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1140960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1140980Z 
2022-11-23T02:25:23.1141073Z Running tests...
2022-11-23T02:25:23.1141359Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1141688Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1141985Z   test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1142219Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50494
2022-11-23T02:25:23.1142456Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50495
2022-11-23T02:25:23.1142850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1143031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1143419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1143618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1144008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1144191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1144594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1144801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1145042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1145284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1145422Z skip: Need at least 8 CUDA devices (3.975s)
2022-11-23T02:25:23.1145460Z 
2022-11-23T02:25:23.1145730Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1145845Z Ran 1 test in 3.975s
2022-11-23T02:25:23.1145865Z 
2022-11-23T02:25:23.1145974Z OK (skipped=1)
2022-11-23T02:25:23.1145993Z 
2022-11-23T02:25:23.1146120Z Generating XML reports...
2022-11-23T02:25:23.1146606Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021655.xml
2022-11-23T02:25:23.1147067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1147268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1147672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1147854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1147892Z 
2022-11-23T02:25:23.1147984Z Running tests...
2022-11-23T02:25:23.1148266Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1148597Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1148911Z   test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1149145Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50693
2022-11-23T02:25:23.1149376Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50694
2022-11-23T02:25:23.1149845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1150027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1150414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1150612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1151003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1151189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1151589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1151792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1152034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1152278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1152361Z ok (5.742s)
2022-11-23T02:25:23.1152382Z 
2022-11-23T02:25:23.1152665Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1152780Z Ran 1 test in 5.743s
2022-11-23T02:25:23.1152799Z 
2022-11-23T02:25:23.1152891Z OK
2022-11-23T02:25:23.1152911Z 
2022-11-23T02:25:23.1153037Z Generating XML reports...
2022-11-23T02:25:23.1153520Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021702.xml
2022-11-23T02:25:23.1153911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1154098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1154511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1154697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1154717Z 
2022-11-23T02:25:23.1154827Z Running tests...
2022-11-23T02:25:23.1155109Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1155442Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1155765Z   test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1155996Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50898
2022-11-23T02:25:23.1156225Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50899
2022-11-23T02:25:23.1156665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1156840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1157271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1157467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1157853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1158035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1158437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1158633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1158877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1159120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1159421Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk3u23b1e
2022-11-23T02:25:23.1159702Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk3u23b1e/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1159971Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpadpn5d3i
2022-11-23T02:25:23.1160252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpadpn5d3i/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1160625Z [W logger.cpp:318] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-11-23T02:25:23.1160872Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1161246Z [W logger.cpp:318] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-11-23T02:25:23.1161492Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1161580Z ok (9.163s)
2022-11-23T02:25:23.1161618Z 
2022-11-23T02:25:23.1161885Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1162091Z Ran 1 test in 9.164s
2022-11-23T02:25:23.1162112Z 
2022-11-23T02:25:23.1162204Z OK
2022-11-23T02:25:23.1162224Z 
2022-11-23T02:25:23.1162354Z Generating XML reports...
2022-11-23T02:25:23.1162834Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021710.xml
2022-11-23T02:25:23.1163208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1163386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1163769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1163949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1163969Z 
2022-11-23T02:25:23.1164077Z Running tests...
2022-11-23T02:25:23.1164346Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1164657Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1164968Z   test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1165189Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51128
2022-11-23T02:25:23.1165407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51129
2022-11-23T02:25:23.1165782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1165996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1166389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1166584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1166947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1167121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1167495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1167686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1167914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1168145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1168390Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo0675xit
2022-11-23T02:25:23.1168715Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo0675xit/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1168973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphf9zucxu
2022-11-23T02:25:23.1169244Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphf9zucxu/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1169480Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1169716Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1169818Z ok (7.378s)
2022-11-23T02:25:23.1169838Z 
2022-11-23T02:25:23.1170109Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1170203Z Ran 1 test in 7.378s
2022-11-23T02:25:23.1170240Z 
2022-11-23T02:25:23.1170313Z OK
2022-11-23T02:25:23.1170331Z 
2022-11-23T02:25:23.1170457Z Generating XML reports...
2022-11-23T02:25:23.1170922Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021722.xml
2022-11-23T02:25:23.1171296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1171475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1171855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1172048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1172068Z 
2022-11-23T02:25:23.1172178Z Running tests...
2022-11-23T02:25:23.1172423Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1172736Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1173057Z   test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1173290Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51349
2022-11-23T02:25:23.1173510Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51350
2022-11-23T02:25:23.1173882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1174058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1174438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1174611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1174982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1175206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1175595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1175789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1176018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1176242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1176502Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz2f2wdy_
2022-11-23T02:25:23.1177025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz2f2wdy_/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1177275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu3cq8zf1
2022-11-23T02:25:23.1177549Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu3cq8zf1/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1177790Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1178105Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1178207Z ok (7.277s)
2022-11-23T02:25:23.1178227Z 
2022-11-23T02:25:23.1178501Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1178613Z Ran 1 test in 7.278s
2022-11-23T02:25:23.1178632Z 
2022-11-23T02:25:23.1178725Z OK
2022-11-23T02:25:23.1178744Z 
2022-11-23T02:25:23.1178849Z Generating XML reports...
2022-11-23T02:25:23.1179313Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021732.xml
2022-11-23T02:25:23.1179686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1179866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1180247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1180443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1180462Z 
2022-11-23T02:25:23.1180570Z Running tests...
2022-11-23T02:25:23.1180833Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1181142Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1181408Z   test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1181629Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51570
2022-11-23T02:25:23.1181848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51571
2022-11-23T02:25:23.1182223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1182399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1182779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1182968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1183336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1183490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1183862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1184049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1184279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1184570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1184683Z ok (22.849s)
2022-11-23T02:25:23.1184702Z 
2022-11-23T02:25:23.1184971Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1185082Z Ran 1 test in 22.849s
2022-11-23T02:25:23.1185102Z 
2022-11-23T02:25:23.1185194Z OK
2022-11-23T02:25:23.1185213Z 
2022-11-23T02:25:23.1185321Z Generating XML reports...
2022-11-23T02:25:23.1185783Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021741.xml
2022-11-23T02:25:23.1186150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1186327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1186706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1186898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1186917Z 
2022-11-23T02:25:23.1187077Z Running tests...
2022-11-23T02:25:23.1187339Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1187633Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1187817Z   test_no_grad (__main__.DistributedDataParallelTest)
2022-11-23T02:25:23.1188068Z Note: this test can be sped up by only running it on a CPU module ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1188289Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51787
2022-11-23T02:25:23.1188504Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51788
2022-11-23T02:25:23.1188877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1189056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1189435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1189627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1189977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1190148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1190525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1190716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1190947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1191171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1191433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfa0yy6kd
2022-11-23T02:25:23.1191706Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfa0yy6kd/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1191940Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0awwln9l
2022-11-23T02:25:23.1192211Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0awwln9l/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1192312Z ok (7.428s)
2022-11-23T02:25:23.1192331Z 
2022-11-23T02:25:23.1192604Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1192716Z Ran 1 test in 7.429s
2022-11-23T02:25:23.1192736Z 
2022-11-23T02:25:23.1192826Z OK
2022-11-23T02:25:23.1192845Z 
2022-11-23T02:25:23.1192968Z Generating XML reports...
2022-11-23T02:25:23.1193433Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021807.xml
2022-11-23T02:25:23.1193854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1194022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1194406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1194597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1194617Z 
2022-11-23T02:25:23.1194726Z Running tests...
2022-11-23T02:25:23.1194987Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1195297Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1195579Z   test_param_layout_mismatch_error (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1195800Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52000
2022-11-23T02:25:23.1196004Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52001
2022-11-23T02:25:23.1196431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1196606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1196967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1197142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1197517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1197706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1198080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1198274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1198491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1198721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1198979Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprflaxybr
2022-11-23T02:25:23.1199250Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprflaxybr/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1199503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmry45j1e
2022-11-23T02:25:23.1199770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmry45j1e/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1199871Z ok (7.018s)
2022-11-23T02:25:23.1199890Z 
2022-11-23T02:25:23.1200158Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1200252Z Ran 1 test in 7.018s
2022-11-23T02:25:23.1200275Z 
2022-11-23T02:25:23.1200368Z OK
2022-11-23T02:25:23.1200387Z 
2022-11-23T02:25:23.1200512Z Generating XML reports...
2022-11-23T02:25:23.1200976Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021817.xml
2022-11-23T02:25:23.1201345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1201519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1201896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1202087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1202107Z 
2022-11-23T02:25:23.1202215Z Running tests...
2022-11-23T02:25:23.1202458Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1202820Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1203092Z   test_pass_default_pg (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1203315Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52213
2022-11-23T02:25:23.1203534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52214
2022-11-23T02:25:23.1203904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1204079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1204459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1204632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1204998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1205170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1205598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1205787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1206016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1206262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.1206486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1206713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.1207115Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1207513Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1207617Z ok (4.000s)
2022-11-23T02:25:23.1207637Z 
2022-11-23T02:25:23.1207905Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1208017Z Ran 1 test in 4.000s
2022-11-23T02:25:23.1208036Z 
2022-11-23T02:25:23.1208127Z OK
2022-11-23T02:25:23.1208145Z 
2022-11-23T02:25:23.1208269Z Generating XML reports...
2022-11-23T02:25:23.1208729Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021826.xml
2022-11-23T02:25:23.1209077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1209252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1209634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1209827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1209849Z 
2022-11-23T02:25:23.1209956Z Running tests...
2022-11-23T02:25:23.1210221Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1210531Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1210812Z   test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1211032Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52416
2022-11-23T02:25:23.1211233Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52417
2022-11-23T02:25:23.1211602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1211776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1212199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1212398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1212761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1212934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1213302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1213471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1213699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1214247Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:25:23.1214536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1215078Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:25:23.1215335Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp700l_1g9
2022-11-23T02:25:23.1215602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp700l_1g9/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1215859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz59s_jlx
2022-11-23T02:25:23.1216126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz59s_jlx/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1216665Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.1217403Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.1217940Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:25:23.1218485Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:25:23.1219017Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.1219614Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.1220157Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:25:23.1220689Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:25:23.1221216Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.1221798Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.1222334Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:25:23.1222866Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:25:23.1223405Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.1223937Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.1224043Z ok (6.921s)
2022-11-23T02:25:23.1224063Z 
2022-11-23T02:25:23.1224343Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1224437Z Ran 1 test in 6.921s
2022-11-23T02:25:23.1224475Z 
2022-11-23T02:25:23.1224547Z OK
2022-11-23T02:25:23.1224566Z 
2022-11-23T02:25:23.1224690Z Generating XML reports...
2022-11-23T02:25:23.1225151Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021833.xml
2022-11-23T02:25:23.1225522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1225698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1226128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1226331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1226350Z 
2022-11-23T02:25:23.1226458Z Running tests...
2022-11-23T02:25:23.1226707Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1227020Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1227314Z   test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1227534Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52637
2022-11-23T02:25:23.1227750Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52638
2022-11-23T02:25:23.1228126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1228303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1228789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1228981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1229325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1229499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1229872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1230060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1230291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1230833Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:25:23.1231066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1231596Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:25:23.1231856Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0e5v8goa
2022-11-23T02:25:23.1232130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0e5v8goa/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1232366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpznv_siyo
2022-11-23T02:25:23.1232637Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpznv_siyo/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1233177Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.1233767Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.1234307Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:25:23.1234856Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:25:23.1235395Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.1235987Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.1236526Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:25:23.1237059Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:25:23.1237591Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.1238115Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.1238653Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:25:23.1239184Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-11-23T02:25:23.1239778Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.1240315Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:25:23.1240420Z ok (7.017s)
2022-11-23T02:25:23.1240440Z 
2022-11-23T02:25:23.1240714Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1240825Z Ran 1 test in 7.017s
2022-11-23T02:25:23.1240844Z 
2022-11-23T02:25:23.1240935Z OK
2022-11-23T02:25:23.1240954Z 
2022-11-23T02:25:23.1241077Z Generating XML reports...
2022-11-23T02:25:23.1241539Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021842.xml
2022-11-23T02:25:23.1241916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1242075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1242510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1242702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1242724Z 
2022-11-23T02:25:23.1242832Z Running tests...
2022-11-23T02:25:23.1243100Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1243411Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1243692Z   test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1243917Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52858
2022-11-23T02:25:23.1244145Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52859
2022-11-23T02:25:23.1244504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1244686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1245065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1245256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1245621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1245801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1246179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1246370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1246586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1246817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1247080Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp88cdbamn
2022-11-23T02:25:23.1247354Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp88cdbamn/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1247612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplb_x84xr
2022-11-23T02:25:23.1247884Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplb_x84xr/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1248124Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1248368Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1248473Z ok (8.391s)
2022-11-23T02:25:23.1248493Z 
2022-11-23T02:25:23.1248789Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1248915Z Ran 1 test in 8.391s
2022-11-23T02:25:23.1248935Z 
2022-11-23T02:25:23.1249031Z OK
2022-11-23T02:25:23.1249051Z 
2022-11-23T02:25:23.1249181Z Generating XML reports...
2022-11-23T02:25:23.1249658Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021852.xml
2022-11-23T02:25:23.1250039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1250221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1250599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1250772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1250814Z 
2022-11-23T02:25:23.1250905Z Running tests...
2022-11-23T02:25:23.1251179Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1251554Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1251839Z   test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1252065Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53079
2022-11-23T02:25:23.1252291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53080
2022-11-23T02:25:23.1252669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1252852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1253203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1253385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1253768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1253968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1254350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1254547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1254784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1255018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1255259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcpa80u0t
2022-11-23T02:25:23.1255535Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcpa80u0t/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1255798Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5pcxzmnw
2022-11-23T02:25:23.1256079Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5pcxzmnw/_remote_module_non_scriptable.py
2022-11-23T02:25:23.1256318Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1256560Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:23.1256666Z ok (7.627s)
2022-11-23T02:25:23.1256686Z 
2022-11-23T02:25:23.1257204Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1257322Z Ran 1 test in 7.627s
2022-11-23T02:25:23.1257342Z 
2022-11-23T02:25:23.1257414Z OK
2022-11-23T02:25:23.1257433Z 
2022-11-23T02:25:23.1257565Z Generating XML reports...
2022-11-23T02:25:23.1258033Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021902.xml
2022-11-23T02:25:23.1258486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1258677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1259066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1259262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1259282Z 
2022-11-23T02:25:23.1259395Z Running tests...
2022-11-23T02:25:23.1259639Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1259957Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1260230Z   test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1260455Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53300
2022-11-23T02:25:23.1260684Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53301
2022-11-23T02:25:23.1260971Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 53302
2022-11-23T02:25:23.1261350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1261531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1261915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1262182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1262564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1262743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1263123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1263317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1263687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1263864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1264244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1264414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1264650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1264884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1265117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:25:23.1265226Z ok (4.116s)
2022-11-23T02:25:23.1265245Z 
2022-11-23T02:25:23.1265514Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1265634Z Ran 1 test in 4.116s
2022-11-23T02:25:23.1265653Z 
2022-11-23T02:25:23.1265750Z OK
2022-11-23T02:25:23.1265769Z 
2022-11-23T02:25:23.1265874Z Generating XML reports...
2022-11-23T02:25:23.1266314Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123021913.xml
2022-11-23T02:25:23.1266690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1266870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1267249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1267444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1267463Z 
2022-11-23T02:25:23.1267632Z Running tests...
2022-11-23T02:25:23.1267908Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1268233Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1268510Z   test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1268738Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53565
2022-11-23T02:25:23.1268960Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53566
2022-11-23T02:25:23.1269180Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 53567
2022-11-23T02:25:23.1269557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1269740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1270114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1270358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1270719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1270915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1271294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1271493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1271860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1272038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1272418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1272610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1272850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1273055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1273283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:25:23.1273390Z ok (16.286s)
2022-11-23T02:25:23.1273409Z 
2022-11-23T02:25:23.1273681Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1273800Z Ran 1 test in 16.286s
2022-11-23T02:25:23.1273819Z 
2022-11-23T02:25:23.1273916Z OK
2022-11-23T02:25:23.1273934Z 
2022-11-23T02:25:23.1274064Z Generating XML reports...
2022-11-23T02:25:23.1274503Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123021919.xml
2022-11-23T02:25:23.1274853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1275038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1275417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1275614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1275633Z 
2022-11-23T02:25:23.1275746Z Running tests...
2022-11-23T02:25:23.1276015Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1276329Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1276754Z   test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest) ... skip: Frequently times out see https://github.com/pytorch/pytorch/issues/58920 (0.001s)
2022-11-23T02:25:23.1276775Z 
2022-11-23T02:25:23.1277099Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1277198Z Ran 1 test in 0.001s
2022-11-23T02:25:23.1277240Z 
2022-11-23T02:25:23.1277330Z OK (skipped=1)
2022-11-23T02:25:23.1277348Z 
2022-11-23T02:25:23.1277475Z Generating XML reports...
2022-11-23T02:25:23.1277919Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123021938.xml
2022-11-23T02:25:23.1278294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1278473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1278854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1279053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1279072Z 
2022-11-23T02:25:23.1279187Z Running tests...
2022-11-23T02:25:23.1279438Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1279812Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1280084Z   test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1280313Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53911
2022-11-23T02:25:23.1280539Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53912
2022-11-23T02:25:23.1280768Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 53913
2022-11-23T02:25:23.1281145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1281323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1281688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1281886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1282262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1282439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1282821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1283017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1283383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1283565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1283938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1284113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1284354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1284587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1284822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:25:23.1284931Z ok (19.119s)
2022-11-23T02:25:23.1284951Z 
2022-11-23T02:25:23.1285226Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1285346Z Ran 1 test in 19.119s
2022-11-23T02:25:23.1285365Z 
2022-11-23T02:25:23.1285461Z OK
2022-11-23T02:25:23.1285480Z 
2022-11-23T02:25:23.1285587Z Generating XML reports...
2022-11-23T02:25:23.1286024Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123021940.xml
2022-11-23T02:25:23.1286457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1286646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1287022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1287214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1287234Z 
2022-11-23T02:25:23.1287347Z Running tests...
2022-11-23T02:25:23.1287611Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1287932Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1288186Z   test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1288411Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54194
2022-11-23T02:25:23.1288640Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54195
2022-11-23T02:25:23.1288864Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 54196
2022-11-23T02:25:23.1289296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1289477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1289856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1290051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1290393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1290569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1290954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1291151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1291520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1291694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1292068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1292266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1292501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:25:23.1292704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1292938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1293043Z ok (18.937s)
2022-11-23T02:25:23.1293066Z 
2022-11-23T02:25:23.1293341Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1293465Z Ran 1 test in 18.937s
2022-11-23T02:25:23.1293484Z 
2022-11-23T02:25:23.1293581Z OK
2022-11-23T02:25:23.1293600Z 
2022-11-23T02:25:23.1293728Z Generating XML reports...
2022-11-23T02:25:23.1294165Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123022002.xml
2022-11-23T02:25:23.1294514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1294695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1295078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1295274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1295294Z 
2022-11-23T02:25:23.1295453Z Running tests...
2022-11-23T02:25:23.1295737Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1296057Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1296329Z   test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1296553Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54477
2022-11-23T02:25:23.1296988Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54478
2022-11-23T02:25:23.1297221Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 54479
2022-11-23T02:25:23.1297604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1297788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1298173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1298375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1298828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1299004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1299359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1299550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1299913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1300088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1300467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1300661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1300900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1301133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1301365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:25:23.1301449Z ok (19.027s)
2022-11-23T02:25:23.1301468Z 
2022-11-23T02:25:23.1301739Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1301855Z Ran 1 test in 19.027s
2022-11-23T02:25:23.1301874Z 
2022-11-23T02:25:23.1301971Z OK
2022-11-23T02:25:23.1301990Z 
2022-11-23T02:25:23.1302120Z Generating XML reports...
2022-11-23T02:25:23.1302557Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123022023.xml
2022-11-23T02:25:23.1302933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1303116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1303472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1303667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1303686Z 
2022-11-23T02:25:23.1303798Z Running tests...
2022-11-23T02:25:23.1304065Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1304385Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1304654Z   test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1304877Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54760
2022-11-23T02:25:23.1305161Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54761
2022-11-23T02:25:23.1305373Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 54762
2022-11-23T02:25:23.1305755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1305936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1306315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1306513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1306884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1307062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1307436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1307633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1308030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1308207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1308579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1308777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1309015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1309244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1309474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:25:23.1309585Z ok (18.968s)
2022-11-23T02:25:23.1309604Z 
2022-11-23T02:25:23.1309853Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1309973Z Ran 1 test in 18.968s
2022-11-23T02:25:23.1309991Z 
2022-11-23T02:25:23.1310087Z OK
2022-11-23T02:25:23.1310106Z 
2022-11-23T02:25:23.1310233Z Generating XML reports...
2022-11-23T02:25:23.1310669Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123022045.xml
2022-11-23T02:25:23.1311040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1311222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1311596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1311786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1311805Z 
2022-11-23T02:25:23.1311899Z Running tests...
2022-11-23T02:25:23.1312169Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1312489Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1312760Z   test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest) ... skip: Test does not pass when run locally (0.001s)
2022-11-23T02:25:23.1312779Z 
2022-11-23T02:25:23.1313045Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1313162Z Ran 1 test in 0.001s
2022-11-23T02:25:23.1313180Z 
2022-11-23T02:25:23.1313292Z OK (skipped=1)
2022-11-23T02:25:23.1313311Z 
2022-11-23T02:25:23.1313437Z Generating XML reports...
2022-11-23T02:25:23.1313851Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123022106.xml
2022-11-23T02:25:23.1314276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1314467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1314855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1315052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1315072Z 
2022-11-23T02:25:23.1315183Z Running tests...
2022-11-23T02:25:23.1315450Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1315766Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1316013Z   test_nccl_timeout (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1316214Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55108
2022-11-23T02:25:23.1316437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55109
2022-11-23T02:25:23.1316666Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 55110
2022-11-23T02:25:23.1317097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1317275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1317660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1317859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1318297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1318454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1318838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1319037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1319403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1319582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1319955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1320147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1320383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1320614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:25:23.1320823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1321318Z [W ProcessGroupNCCL.cpp:950] [Rank 2] Found key in store: NCCLABORTEDCOMM:20bacfac1102000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, from rank: 0. This means that rank has aborted its NCCL communicators previously and is not in a healthy state.. Aborting appropriate communicators
2022-11-23T02:25:23.1321812Z [W ProcessGroupNCCL.cpp:950] [Rank 1] Found key in store: NCCLABORTEDCOMM:20bacfac1102000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, from rank: 0. This means that rank has aborted its NCCL communicators previously and is not in a healthy state.. Aborting appropriate communicators
2022-11-23T02:25:23.1321919Z ok (27.268s)
2022-11-23T02:25:23.1321940Z 
2022-11-23T02:25:23.1322215Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1322336Z Ran 1 test in 27.269s
2022-11-23T02:25:23.1322355Z 
2022-11-23T02:25:23.1322452Z OK
2022-11-23T02:25:23.1322470Z 
2022-11-23T02:25:23.1322597Z Generating XML reports...
2022-11-23T02:25:23.1323086Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123022109.xml
2022-11-23T02:25:23.1323480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1323659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1324020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1324217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1324236Z 
2022-11-23T02:25:23.1324348Z Running tests...
2022-11-23T02:25:23.1324621Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1324939Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1325274Z   test_allgather_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1325500Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55400
2022-11-23T02:25:23.1325925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1326082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1326460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1326652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1326889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1327137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.1327545Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:25:23.1327653Z ok (5.516s)
2022-11-23T02:25:23.1327673Z 
2022-11-23T02:25:23.1327949Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1328068Z Ran 1 test in 5.516s
2022-11-23T02:25:23.1328088Z 
2022-11-23T02:25:23.1328161Z OK
2022-11-23T02:25:23.1328180Z 
2022-11-23T02:25:23.1328311Z Generating XML reports...
2022-11-23T02:25:23.1328870Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123022138.xml
2022-11-23T02:25:23.1329246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1329429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1329813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1330014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1330033Z 
2022-11-23T02:25:23.1330146Z Running tests...
2022-11-23T02:25:23.1330394Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1330710Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1331051Z   test_allreduce_coalesced (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1331276Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55539
2022-11-23T02:25:23.1331651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1331831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1332213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1332457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1332697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1332930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.1333340Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:25:23.1334087Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:25:23.1334209Z   warnings.warn(
2022-11-23T02:25:23.1334316Z ok (5.465s)
2022-11-23T02:25:23.1334336Z 
2022-11-23T02:25:23.1334605Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1334723Z Ran 1 test in 5.465s
2022-11-23T02:25:23.1334746Z 
2022-11-23T02:25:23.1334842Z OK
2022-11-23T02:25:23.1334861Z 
2022-11-23T02:25:23.1335054Z Generating XML reports...
2022-11-23T02:25:23.1335592Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123022146.xml
2022-11-23T02:25:23.1335964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1336143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1336528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1336937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1336959Z 
2022-11-23T02:25:23.1337082Z Running tests...
2022-11-23T02:25:23.1337363Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1337685Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1337999Z   test_collectives (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1338228Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55678
2022-11-23T02:25:23.1338610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1338791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1339176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1339373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1339611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1339870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.1340276Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:25:23.1340364Z ok (5.489s)
2022-11-23T02:25:23.1340384Z 
2022-11-23T02:25:23.1340654Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1340770Z Ran 1 test in 5.490s
2022-11-23T02:25:23.1340789Z 
2022-11-23T02:25:23.1340888Z OK
2022-11-23T02:25:23.1340907Z 
2022-11-23T02:25:23.1341033Z Generating XML reports...
2022-11-23T02:25:23.1341593Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123022154.xml
2022-11-23T02:25:23.1341969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1342149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1342616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1342805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1342829Z 
2022-11-23T02:25:23.1342941Z Running tests...
2022-11-23T02:25:23.1343215Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1343532Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1343872Z   test_reduce_scatter_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1344097Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55817
2022-11-23T02:25:23.1344470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1344650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1345012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1345273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1345504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1345756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.1346166Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:25:23.1346276Z ok (5.635s)
2022-11-23T02:25:23.1346296Z 
2022-11-23T02:25:23.1346563Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1346678Z Ran 1 test in 5.635s
2022-11-23T02:25:23.1346697Z 
2022-11-23T02:25:23.1346794Z OK
2022-11-23T02:25:23.1346813Z 
2022-11-23T02:25:23.1346919Z Generating XML reports...
2022-11-23T02:25:23.1347478Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123022202.xml
2022-11-23T02:25:23.1347859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1348041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1348426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1348624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1348643Z 
2022-11-23T02:25:23.1348758Z Running tests...
2022-11-23T02:25:23.1349026Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1349317Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1349594Z   test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest) ... skip: GPUs are available, skipping test (0.001s)
2022-11-23T02:25:23.1349614Z 
2022-11-23T02:25:23.1349886Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1349999Z Ran 1 test in 0.001s
2022-11-23T02:25:23.1350019Z 
2022-11-23T02:25:23.1350129Z OK (skipped=1)
2022-11-23T02:25:23.1350148Z 
2022-11-23T02:25:23.1350276Z Generating XML reports...
2022-11-23T02:25:23.1350733Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLNoGPUTest-20221123022210.xml
2022-11-23T02:25:23.1351106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1351287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1351647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1351887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1351908Z 
2022-11-23T02:25:23.1352025Z Running tests...
2022-11-23T02:25:23.1352296Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1352613Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1352877Z   test_allgather_base_basics (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1353101Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56021
2022-11-23T02:25:23.1353328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56022
2022-11-23T02:25:23.1353700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1353860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1354250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1354448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1354874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1355055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1355433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1355631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1355868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1356096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.1356329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1356580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.1356990Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1357393Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1357502Z ok (5.663s)
2022-11-23T02:25:23.1357523Z 
2022-11-23T02:25:23.1357798Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1357913Z Ran 1 test in 5.663s
2022-11-23T02:25:23.1357934Z 
2022-11-23T02:25:23.1358034Z OK
2022-11-23T02:25:23.1358053Z 
2022-11-23T02:25:23.1358160Z Generating XML reports...
2022-11-23T02:25:23.1358601Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022213.xml
2022-11-23T02:25:23.1358982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1359163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1359549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1359749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1359768Z 
2022-11-23T02:25:23.1359880Z Running tests...
2022-11-23T02:25:23.1360146Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1360442Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1360698Z   test_allgather_base_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1360923Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56226
2022-11-23T02:25:23.1361147Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56227
2022-11-23T02:25:23.1361571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1361759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1362223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1362422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1362797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1362954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1363329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1363521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1363762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1364015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.1364305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1364549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.1364957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1365361Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1365444Z ok (7.021s)
2022-11-23T02:25:23.1365463Z 
2022-11-23T02:25:23.1365734Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1365850Z Ran 1 test in 7.021s
2022-11-23T02:25:23.1365868Z 
2022-11-23T02:25:23.1365969Z OK
2022-11-23T02:25:23.1365989Z 
2022-11-23T02:25:23.1366118Z Generating XML reports...
2022-11-23T02:25:23.1366555Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022221.xml
2022-11-23T02:25:23.1366926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1367106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1367467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1367664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1367683Z 
2022-11-23T02:25:23.1367796Z Running tests...
2022-11-23T02:25:23.1368067Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1368381Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1368634Z   test_allgather_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1368867Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56439
2022-11-23T02:25:23.1369094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56440
2022-11-23T02:25:23.1369444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1369626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1370007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1370201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1370570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1370837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1371231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1371427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1371663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1371889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.1372117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1372362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.1372770Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1373172Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1373279Z ok (6.925s)
2022-11-23T02:25:23.1373350Z 
2022-11-23T02:25:23.1373626Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1373742Z Ran 1 test in 6.925s
2022-11-23T02:25:23.1373764Z 
2022-11-23T02:25:23.1373863Z OK
2022-11-23T02:25:23.1373881Z 
2022-11-23T02:25:23.1373986Z Generating XML reports...
2022-11-23T02:25:23.1374425Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022230.xml
2022-11-23T02:25:23.1374801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1374984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1375366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1375565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1375584Z 
2022-11-23T02:25:23.1375700Z Running tests...
2022-11-23T02:25:23.1375969Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1376263Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1376509Z   test_allreduce_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1376984Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56652
2022-11-23T02:25:23.1377220Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56653
2022-11-23T02:25:23.1377606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1377787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1378174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1378373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1378754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1378910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1379290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1379483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1379717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1379970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.1380205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1380526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.1380955Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1381336Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1381443Z ok (6.787s)
2022-11-23T02:25:23.1381463Z 
2022-11-23T02:25:23.1381733Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1381852Z Ran 1 test in 6.787s
2022-11-23T02:25:23.1381871Z 
2022-11-23T02:25:23.1381968Z OK
2022-11-23T02:25:23.1381987Z 
2022-11-23T02:25:23.1382115Z Generating XML reports...
2022-11-23T02:25:23.1382554Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022240.xml
2022-11-23T02:25:23.1382932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1383112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1383543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1383742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1383761Z 
2022-11-23T02:25:23.1383877Z Running tests...
2022-11-23T02:25:23.1384148Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1384465Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1384698Z   test_barrier (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1384922Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56865
2022-11-23T02:25:23.1385148Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56866
2022-11-23T02:25:23.1385506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1385692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1386077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1386273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1386641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1386821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1387200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1387395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1387633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1387865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.1388096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1388338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.1388747Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1389142Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1389251Z ok (6.932s)
2022-11-23T02:25:23.1389270Z 
2022-11-23T02:25:23.1389539Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1389657Z Ran 1 test in 6.933s
2022-11-23T02:25:23.1389675Z 
2022-11-23T02:25:23.1389798Z OK
2022-11-23T02:25:23.1389847Z 
2022-11-23T02:25:23.1389956Z Generating XML reports...
2022-11-23T02:25:23.1390408Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022249.xml
2022-11-23T02:25:23.1390778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1390954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1391331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1391521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1391541Z 
2022-11-23T02:25:23.1391651Z Running tests...
2022-11-23T02:25:23.1391913Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1392215Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1392459Z   test_broadcast_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1392730Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57078
2022-11-23T02:25:23.1392949Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57079
2022-11-23T02:25:23.1393321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1393497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1393872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1394067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1394413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1394593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1394966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1395158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1395388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1395632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.1395861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1396105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.1396506Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1396888Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1396993Z ok (6.910s)
2022-11-23T02:25:23.1397012Z 
2022-11-23T02:25:23.1397277Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1397391Z Ran 1 test in 6.910s
2022-11-23T02:25:23.1397410Z 
2022-11-23T02:25:23.1397503Z OK
2022-11-23T02:25:23.1397522Z 
2022-11-23T02:25:23.1397646Z Generating XML reports...
2022-11-23T02:25:23.1398143Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022259.xml
2022-11-23T02:25:23.1398513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1398691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1399055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1399295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1399320Z 
2022-11-23T02:25:23.1399434Z Running tests...
2022-11-23T02:25:23.1399698Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1400011Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1400252Z   test_empty_tensors (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1400472Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57291
2022-11-23T02:25:23.1400690Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57292
2022-11-23T02:25:23.1401041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1401217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1401596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1401852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1402220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1402393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1402764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1402954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1403167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1403412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.1403642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1403878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.1404284Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1404679Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1404780Z ok (6.789s)
2022-11-23T02:25:23.1404799Z 
2022-11-23T02:25:23.1405062Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1405173Z Ran 1 test in 6.790s
2022-11-23T02:25:23.1405193Z 
2022-11-23T02:25:23.1405267Z OK
2022-11-23T02:25:23.1405307Z 
2022-11-23T02:25:23.1405412Z Generating XML reports...
2022-11-23T02:25:23.1405843Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022308.xml
2022-11-23T02:25:23.1406219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1406400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1406783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1406973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1406996Z 
2022-11-23T02:25:23.1407103Z Running tests...
2022-11-23T02:25:23.1407365Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1407660Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1407910Z   test_gather_checks (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1408129Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57504
2022-11-23T02:25:23.1408396Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57505
2022-11-23T02:25:23.1408774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1408953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1409331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1409521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1409871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1410046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1410417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1410606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1410842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1411224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.1411449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1411698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.1412100Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1412477Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1412580Z ok (5.722s)
2022-11-23T02:25:23.1412599Z 
2022-11-23T02:25:23.1412861Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1412975Z Ran 1 test in 5.722s
2022-11-23T02:25:23.1412998Z 
2022-11-23T02:25:23.1413090Z OK
2022-11-23T02:25:23.1413109Z 
2022-11-23T02:25:23.1413236Z Generating XML reports...
2022-11-23T02:25:23.1413668Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022317.xml
2022-11-23T02:25:23.1414037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1414194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1414576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1414769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1414788Z 
2022-11-23T02:25:23.1414895Z Running tests...
2022-11-23T02:25:23.1415158Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1415474Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1415712Z   test_gather_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1415936Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57709
2022-11-23T02:25:23.1416155Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57710
2022-11-23T02:25:23.1416509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1416686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1417332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1417524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1417892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1418159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1418549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1418739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1418954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1419197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.1419425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1419670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.1420069Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1420469Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1420641Z ok (6.970s)
2022-11-23T02:25:23.1420660Z 
2022-11-23T02:25:23.1420934Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1421049Z Ran 1 test in 6.970s
2022-11-23T02:25:23.1421068Z 
2022-11-23T02:25:23.1421140Z OK
2022-11-23T02:25:23.1421159Z 
2022-11-23T02:25:23.1421292Z Generating XML reports...
2022-11-23T02:25:23.1421727Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022325.xml
2022-11-23T02:25:23.1422100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1422281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1422669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1422865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1422889Z 
2022-11-23T02:25:23.1423004Z Running tests...
2022-11-23T02:25:23.1423274Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1423570Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1423817Z   test_gather_stress (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1424037Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57926
2022-11-23T02:25:23.1424254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57927
2022-11-23T02:25:23.1424624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1424802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1425182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1425377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1425723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1425895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1426269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1426461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1426692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1426938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.1427213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1427458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.1427867Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1428247Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1428348Z ok (11.490s)
2022-11-23T02:25:23.1428367Z 
2022-11-23T02:25:23.1428632Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1428747Z Ran 1 test in 11.490s
2022-11-23T02:25:23.1428766Z 
2022-11-23T02:25:23.1428859Z OK
2022-11-23T02:25:23.1428878Z 
2022-11-23T02:25:23.1429003Z Generating XML reports...
2022-11-23T02:25:23.1429433Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022335.xml
2022-11-23T02:25:23.1429807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1430020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1430402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1430593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1430613Z 
2022-11-23T02:25:23.1430720Z Running tests...
2022-11-23T02:25:23.1430983Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1431292Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1431551Z   test_nccl_dist_backend_error (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1431771Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58143
2022-11-23T02:25:23.1431998Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58144
2022-11-23T02:25:23.1432354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1432530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1432907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1433098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1433461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1433635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1434008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1434201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1434414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1434663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.1434890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1435132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.1435534Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1435932Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1436035Z ok (5.730s)
2022-11-23T02:25:23.1436054Z 
2022-11-23T02:25:23.1436339Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1436498Z Ran 1 test in 5.730s
2022-11-23T02:25:23.1436522Z 
2022-11-23T02:25:23.1436597Z OK
2022-11-23T02:25:23.1436619Z 
2022-11-23T02:25:23.1436743Z Generating XML reports...
2022-11-23T02:25:23.1437177Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022349.xml
2022-11-23T02:25:23.1437544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1437719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1438094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1438286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1438305Z 
2022-11-23T02:25:23.1438413Z Running tests...
2022-11-23T02:25:23.1438655Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1438974Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1439265Z   test_reduce_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1439486Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58351
2022-11-23T02:25:23.1439705Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58352
2022-11-23T02:25:23.1440077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1440256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1440634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1440824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1441178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1441358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1441740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1441933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1442170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1442417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.1442649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1442898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.1443285Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1443688Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1443800Z ok (6.861s)
2022-11-23T02:25:23.1443818Z 
2022-11-23T02:25:23.1444090Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1444205Z Ran 1 test in 6.861s
2022-11-23T02:25:23.1444224Z 
2022-11-23T02:25:23.1444320Z OK
2022-11-23T02:25:23.1444339Z 
2022-11-23T02:25:23.1444470Z Generating XML reports...
2022-11-23T02:25:23.1444903Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022357.xml
2022-11-23T02:25:23.1445277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1445434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1445869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1446074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1446096Z 
2022-11-23T02:25:23.1446210Z Running tests...
2022-11-23T02:25:23.1446479Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1446795Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1447065Z   test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1447289Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58564
2022-11-23T02:25:23.1447490Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58565
2022-11-23T02:25:23.1447865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1448048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1448431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1448679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1449047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1449226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1449609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1449802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1450015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1450265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.1450497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1450748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.1451153Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1451554Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1451660Z ok (5.556s)
2022-11-23T02:25:23.1451679Z 
2022-11-23T02:25:23.1451948Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1452063Z Ran 1 test in 5.556s
2022-11-23T02:25:23.1452083Z 
2022-11-23T02:25:23.1452157Z OK
2022-11-23T02:25:23.1452175Z 
2022-11-23T02:25:23.1452304Z Generating XML reports...
2022-11-23T02:25:23.1452743Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022406.xml
2022-11-23T02:25:23.1453119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1453303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1453687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1453881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1453901Z 
2022-11-23T02:25:23.1454013Z Running tests...
2022-11-23T02:25:23.1454257Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1454579Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1454842Z   test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1455113Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58769
2022-11-23T02:25:23.1455396Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58770
2022-11-23T02:25:23.1455774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1455954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1456339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1456540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1457074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1457266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1457654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1457854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1458169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1458417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.1458651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1458900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.1459290Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1459695Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1459801Z ok (6.770s)
2022-11-23T02:25:23.1459820Z 
2022-11-23T02:25:23.1460092Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1460208Z Ran 1 test in 6.770s
2022-11-23T02:25:23.1460230Z 
2022-11-23T02:25:23.1460326Z OK
2022-11-23T02:25:23.1460345Z 
2022-11-23T02:25:23.1460474Z Generating XML reports...
2022-11-23T02:25:23.1460912Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022414.xml
2022-11-23T02:25:23.1461285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1461443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1461830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1462025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1462045Z 
2022-11-23T02:25:23.1462244Z Running tests...
2022-11-23T02:25:23.1462530Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1462846Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1463110Z   test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1463337Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58982
2022-11-23T02:25:23.1463539Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58983
2022-11-23T02:25:23.1463915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1464094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1464479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1464675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1465109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1465299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1465680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1465875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1466089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1466338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.1466569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1466814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.1467221Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1467619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1467792Z ok (7.017s)
2022-11-23T02:25:23.1467811Z 
2022-11-23T02:25:23.1468081Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1468195Z Ran 1 test in 7.017s
2022-11-23T02:25:23.1468216Z 
2022-11-23T02:25:23.1468288Z OK
2022-11-23T02:25:23.1468307Z 
2022-11-23T02:25:23.1468435Z Generating XML reports...
2022-11-23T02:25:23.1468869Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022424.xml
2022-11-23T02:25:23.1469239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1469424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1469816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1470018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1470037Z 
2022-11-23T02:25:23.1470148Z Running tests...
2022-11-23T02:25:23.1470393Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1470707Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1470960Z   test_scatter_checks (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1471187Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59195
2022-11-23T02:25:23.1471412Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59196
2022-11-23T02:25:23.1471788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1471972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1472359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1472556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1472905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1473084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1473462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1473661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1473898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1474193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.1474428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1474679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.1475064Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1475464Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1475569Z ok (5.610s)
2022-11-23T02:25:23.1475588Z 
2022-11-23T02:25:23.1475856Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1475975Z Ran 1 test in 5.615s
2022-11-23T02:25:23.1475994Z 
2022-11-23T02:25:23.1476090Z OK
2022-11-23T02:25:23.1476109Z 
2022-11-23T02:25:23.1476237Z Generating XML reports...
2022-11-23T02:25:23.1476676Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022433.xml
2022-11-23T02:25:23.1477049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1477258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1477643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1477839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1477859Z 
2022-11-23T02:25:23.1477970Z Running tests...
2022-11-23T02:25:23.1478237Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1478549Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1478793Z   test_scatter_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1479027Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59400
2022-11-23T02:25:23.1479228Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59401
2022-11-23T02:25:23.1479608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1479790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1480172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1480368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1480736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1480914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1481296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1481491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1481707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1481956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.1482191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1482436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.1482840Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1483241Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1483347Z ok (6.898s)
2022-11-23T02:25:23.1483366Z 
2022-11-23T02:25:23.1483683Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1483783Z Ran 1 test in 6.898s
2022-11-23T02:25:23.1483831Z 
2022-11-23T02:25:23.1483904Z OK
2022-11-23T02:25:23.1483923Z 
2022-11-23T02:25:23.1484055Z Generating XML reports...
2022-11-23T02:25:23.1484492Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022441.xml
2022-11-23T02:25:23.1484860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1485042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1485423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1485618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1485638Z 
2022-11-23T02:25:23.1485750Z Running tests...
2022-11-23T02:25:23.1486000Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1486317Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1486619Z   test_scatter_stress (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1486845Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59617
2022-11-23T02:25:23.1487069Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59618
2022-11-23T02:25:23.1487442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1487620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1488004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1488175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1488548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1488729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1489107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1489302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1489536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1489785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.1490013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1490256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.1490641Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1491045Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1491152Z ok (11.522s)
2022-11-23T02:25:23.1491171Z 
2022-11-23T02:25:23.1491439Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1491557Z Ran 1 test in 11.522s
2022-11-23T02:25:23.1491576Z 
2022-11-23T02:25:23.1491676Z OK
2022-11-23T02:25:23.1491694Z 
2022-11-23T02:25:23.1491826Z Generating XML reports...
2022-11-23T02:25:23.1492259Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022451.xml
2022-11-23T02:25:23.1492632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1492790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1493230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1493437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1493456Z 
2022-11-23T02:25:23.1493571Z Running tests...
2022-11-23T02:25:23.1493846Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1494166Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1494406Z   test_send_recv (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1494632Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59834
2022-11-23T02:25:23.1494832Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59835
2022-11-23T02:25:23.1495212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1495396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1495821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1496001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1496382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1496579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1497159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1497357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1497570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:23.1497827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.1498060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:23.1498306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:23.1498713Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1499108Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:23.1499219Z ok (5.707s)
2022-11-23T02:25:23.1499238Z 
2022-11-23T02:25:23.1499507Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1499601Z Ran 1 test in 5.707s
2022-11-23T02:25:23.1499644Z 
2022-11-23T02:25:23.1499717Z OK
2022-11-23T02:25:23.1499736Z 
2022-11-23T02:25:23.1499867Z Generating XML reports...
2022-11-23T02:25:23.1500313Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022505.xml
2022-11-23T02:25:23.1500689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1500869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1501254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1501454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1501474Z 
2022-11-23T02:25:23.1501585Z Running tests...
2022-11-23T02:25:23.1501830Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1502143Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1502377Z   test_common_errors (__main__.RendezvousEnvTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1502703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.1503125Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:25:23.1503374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.1503767Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:25:23.1504013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.1504414Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:25:23.1504633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.1505034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:25:23.1505142Z ok (1.747s)
2022-11-23T02:25:23.1505220Z 
2022-11-23T02:25:23.1505496Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1505616Z Ran 1 test in 1.747s
2022-11-23T02:25:23.1505635Z 
2022-11-23T02:25:23.1505731Z OK
2022-11-23T02:25:23.1505750Z 
2022-11-23T02:25:23.1505880Z Generating XML reports...
2022-11-23T02:25:23.1506306Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-RendezvousEnvTest-20221123022513.xml
2022-11-23T02:25:23.1506660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:23.1506840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:23.1507225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:23.1507426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:23.1507445Z 
2022-11-23T02:25:23.1507560Z Running tests...
2022-11-23T02:25:23.1507826Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1508140Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T02:25:23.1508385Z   test_default_store_timeout_nccl (__main__.TimeoutTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:23.1508631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.1509012Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:25:23.1509259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:23.1509664Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:25:23.1509773Z ok (4.743s)
2022-11-23T02:25:23.1509793Z 
2022-11-23T02:25:23.1510061Z ----------------------------------------------------------------------
2022-11-23T02:25:23.1510176Z Ran 1 test in 4.743s
2022-11-23T02:25:23.1510195Z 
2022-11-23T02:25:23.1510294Z OK
2022-11-23T02:25:23.1510313Z 
2022-11-23T02:25:23.1510443Z Generating XML reports...
2022-11-23T02:25:23.1510823Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-TimeoutTest-20221123022517.xml
2022-11-23T02:25:23.1510865Z 
2022-11-23T02:25:23.1511447Z ##[endgroup]
2022-11-23T02:25:23.1511889Z FINISHED PRINTING LOG FILE of distributed/test_c10d_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_nccl_hnzo7h72)
2022-11-23T02:25:23.1511911Z 
2022-11-23T02:25:23.4207703Z 
2022-11-23T02:25:23.4208405Z real	20m15.396s
2022-11-23T02:25:23.4208700Z user	31m25.766s
2022-11-23T02:25:23.4208941Z sys	24m21.479s
2022-11-23T02:25:23.4209802Z + python test/run_test.py --verbose -i distributed/test_c10d_spawn_gloo
2022-11-23T02:25:25.7826581Z Ignoring disabled issues:  []
2022-11-23T02:25:25.8357229Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T02:25:25.8357909Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T02:25:25.8358282Z Selected tests:
2022-11-23T02:25:25.8358574Z  distributed/test_c10d_spawn_gloo
2022-11-23T02:25:25.8387299Z Prioritized test from test file changes.
2022-11-23T02:25:25.8387877Z reordering tests for PR:
2022-11-23T02:25:25.8388177Z prioritized: []
2022-11-23T02:25:25.8388671Z the rest: ['distributed/test_c10d_spawn_gloo']
2022-11-23T02:25:25.8388888Z 
2022-11-23T02:25:25.8389436Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T02:25:25.8390407Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T02:25:25.8396873Z parallel (file granularity) tests:
2022-11-23T02:25:25.8397373Z  
2022-11-23T02:25:25.8397651Z serial (file granularity) tests:
2022-11-23T02:25:25.8397958Z  distributed/test_c10d_spawn_gloo
2022-11-23T02:25:28.1690315Z Ignoring disabled issues:  []
2022-11-23T02:25:28.1772808Z Ignoring disabled issues:  []
2022-11-23T02:25:28.6413999Z Running distributed/test_c10d_spawn_gloo ... [2022-11-23 02:25:28.640727]
2022-11-23T02:25:28.6414833Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_spawn_gloo.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:25:28.641179]
2022-11-23T02:27:04.4480089Z 
2022-11-23T02:27:04.4484498Z Expand the folded group to see the log file of distributed/test_c10d_spawn_gloo
2022-11-23T02:27:04.4485556Z ##[group]PRINTING LOG FILE of distributed/test_c10d_spawn_gloo (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_gloo_z95s6kkl)
2022-11-23T02:27:04.4486147Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp39djvxmd
2022-11-23T02:27:04.4486712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp39djvxmd/_remote_module_non_scriptable.py
2022-11-23T02:27:04.4488449Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:27:04.4489230Z <unittest.suite.TestSuite tests=[<__main__.DistributedDataParallelSingleProcessTest testMethod=test_cpu>, <__main__.DistributedDataParallelSingleProcessTest testMethod=test_cuda>, <__main__.DistributedDataParallelSingleProcessTest testMethod=test_rnn>]>
2022-11-23T02:27:04.4490105Z test_cpu (__main__.DistributedDataParallelSingleProcessTest)
2022-11-23T02:27:04.4490572Z test_cuda (__main__.DistributedDataParallelSingleProcessTest)
2022-11-23T02:27:04.4492272Z test_rnn (__main__.DistributedDataParallelSingleProcessTest)
2022-11-23T02:27:04.4492661Z <unittest.suite.TestSuite tests=[]>
2022-11-23T02:27:04.4492996Z <unittest.suite.TestSuite tests=[]>
2022-11-23T02:27:04.4494166Z <unittest.suite.TestSuite tests=[<__main__.TestDistributedNNFunctionsGloo testMethod=test_all_gather>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_gather>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_scatter>]>
2022-11-23T02:27:04.4495330Z test_all_gather (__main__.TestDistributedNNFunctionsGloo)
2022-11-23T02:27:04.4503429Z test_all_to_all (__main__.TestDistributedNNFunctionsGloo)
2022-11-23T02:27:04.4503863Z test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo)
2022-11-23T02:27:04.4504520Z test_allreduce (__main__.TestDistributedNNFunctionsGloo)
2022-11-23T02:27:04.4504970Z test_broadcast (__main__.TestDistributedNNFunctionsGloo)
2022-11-23T02:27:04.4505364Z test_gather (__main__.TestDistributedNNFunctionsGloo)
2022-11-23T02:27:04.4505743Z test_reduce (__main__.TestDistributedNNFunctionsGloo)
2022-11-23T02:27:04.4506145Z test_scatter (__main__.TestDistributedNNFunctionsGloo)
2022-11-23T02:27:04.4506881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:04.4507352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:04.4507953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:04.4508431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:04.4508876Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0yq0w5pc
2022-11-23T02:27:04.4509456Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0yq0w5pc/_remote_module_non_scriptable.py
2022-11-23T02:27:04.4510013Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:27:04.4510243Z 
2022-11-23T02:27:04.4510363Z Running tests...
2022-11-23T02:27:04.4510757Z ----------------------------------------------------------------------
2022-11-23T02:27:04.4511332Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-11-23T02:27:04.4512088Z   test_cpu (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:27:04.4512556Z ok (0.026s)
2022-11-23T02:27:04.4512687Z 
2022-11-23T02:27:04.4512962Z ----------------------------------------------------------------------
2022-11-23T02:27:04.4513283Z Ran 1 test in 0.026s
2022-11-23T02:27:04.4513457Z 
2022-11-23T02:27:04.4513540Z OK
2022-11-23T02:27:04.4513677Z 
2022-11-23T02:27:04.4513795Z Generating XML reports...
2022-11-23T02:27:04.4514503Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20221123022535.xml
2022-11-23T02:27:04.4515512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:04.4515960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:04.4516513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:04.4516985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:04.4517457Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5lan_4is
2022-11-23T02:27:04.4517981Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5lan_4is/_remote_module_non_scriptable.py
2022-11-23T02:27:04.4518410Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:27:04.4518609Z 
2022-11-23T02:27:04.4518726Z Running tests...
2022-11-23T02:27:04.4519143Z ----------------------------------------------------------------------
2022-11-23T02:27:04.4519672Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-11-23T02:27:04.4520294Z   test_cuda (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:27:04.4520766Z ok (0.503s)
2022-11-23T02:27:04.4520915Z 
2022-11-23T02:27:04.4521188Z ----------------------------------------------------------------------
2022-11-23T02:27:04.4521496Z Ran 1 test in 0.503s
2022-11-23T02:27:04.4521658Z 
2022-11-23T02:27:04.4521751Z OK
2022-11-23T02:27:04.4521886Z 
2022-11-23T02:27:04.4522012Z Generating XML reports...
2022-11-23T02:27:04.4522688Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20221123022540.xml
2022-11-23T02:27:04.4523554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:04.4524015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:04.4524601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:04.4525057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:04.4525526Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgb2x8323
2022-11-23T02:27:04.4526068Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgb2x8323/_remote_module_non_scriptable.py
2022-11-23T02:27:04.4526478Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:27:04.4526678Z 
2022-11-23T02:27:04.4526788Z Running tests...
2022-11-23T02:27:04.4527199Z ----------------------------------------------------------------------
2022-11-23T02:27:04.4527747Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-11-23T02:27:04.4528346Z   test_rnn (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:27:04.4528884Z ok (1.315s)
2022-11-23T02:27:04.4529037Z 
2022-11-23T02:27:04.4529305Z ----------------------------------------------------------------------
2022-11-23T02:27:04.4529616Z Ran 1 test in 1.316s
2022-11-23T02:27:04.4529780Z 
2022-11-23T02:27:04.4529874Z OK
2022-11-23T02:27:04.4530010Z 
2022-11-23T02:27:04.4530137Z Generating XML reports...
2022-11-23T02:27:04.4530833Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20221123022544.xml
2022-11-23T02:27:04.4531597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:04.4532052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:04.4532640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:04.4533099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:04.4533576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprqkgzzqn
2022-11-23T02:27:04.4534124Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprqkgzzqn/_remote_module_non_scriptable.py
2022-11-23T02:27:04.4534557Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:27:04.4534738Z 
2022-11-23T02:27:04.4534848Z Running tests...
2022-11-23T02:27:04.4535260Z ----------------------------------------------------------------------
2022-11-23T02:27:04.4535805Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-11-23T02:27:04.4536359Z   test_all_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60704
2022-11-23T02:27:04.4537293Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60705
2022-11-23T02:27:04.4537914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:04.4538436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:04.4539006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:04.4539468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:04.4540047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:04.4540498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:04.4541053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:04.4541628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:04.4542115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvjys8yx6
2022-11-23T02:27:04.4542646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvjys8yx6/_remote_module_non_scriptable.py
2022-11-23T02:27:04.4543187Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptf41v1qp
2022-11-23T02:27:04.4543723Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptf41v1qp/_remote_module_non_scriptable.py
2022-11-23T02:27:04.4544148Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:27:04.4544534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:04.4544929Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:27:04.4545330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:04.4545801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:04.4546299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:04.4547057Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:04.4547747Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:04.4548121Z ok (5.490s)
2022-11-23T02:27:04.4548268Z 
2022-11-23T02:27:04.4548531Z ----------------------------------------------------------------------
2022-11-23T02:27:04.4548860Z Ran 1 test in 5.490s
2022-11-23T02:27:04.4549020Z 
2022-11-23T02:27:04.4549093Z OK
2022-11-23T02:27:04.4549228Z 
2022-11-23T02:27:04.4549354Z Generating XML reports...
2022-11-23T02:27:04.4549998Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123022550.xml
2022-11-23T02:27:04.4550747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:04.4551186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:04.4551765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:04.4552241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:04.4552690Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ufpzr8f
2022-11-23T02:27:04.4553241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ufpzr8f/_remote_module_non_scriptable.py
2022-11-23T02:27:04.4553671Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:27:04.4553866Z 
2022-11-23T02:27:04.4553979Z Running tests...
2022-11-23T02:27:04.4554368Z ----------------------------------------------------------------------
2022-11-23T02:27:04.4554915Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-11-23T02:27:04.4555492Z   test_all_to_all (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60920
2022-11-23T02:27:04.4556016Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60921
2022-11-23T02:27:04.4556633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:04.4557083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:04.4557656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:04.4558107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:04.4558682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:04.4559186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:04.4559773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:04.4560223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:04.4560693Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzn4ekdj1
2022-11-23T02:27:04.4561238Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzn4ekdj1/_remote_module_non_scriptable.py
2022-11-23T02:27:04.4561754Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpauthc17o
2022-11-23T02:27:04.4562291Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpauthc17o/_remote_module_non_scriptable.py
2022-11-23T02:27:04.4562718Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:27:04.4563041Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:27:04.4563430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:04.4563903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:04.4564515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:04.4564996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:04.4565662Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:04.4566350Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:04.4566747Z ok (5.586s)
2022-11-23T02:27:04.4566880Z 
2022-11-23T02:27:04.4567152Z ----------------------------------------------------------------------
2022-11-23T02:27:04.4567480Z Ran 1 test in 5.586s
2022-11-23T02:27:04.4567642Z 
2022-11-23T02:27:04.4567734Z OK
2022-11-23T02:27:04.4567868Z 
2022-11-23T02:27:04.4567978Z Generating XML reports...
2022-11-23T02:27:04.4568630Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123022559.xml
2022-11-23T02:27:04.4569377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:04.4569831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:04.4570390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:04.4570861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:04.4571330Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpftby6f9t
2022-11-23T02:27:04.4571874Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpftby6f9t/_remote_module_non_scriptable.py
2022-11-23T02:27:04.4572284Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:27:04.4572486Z 
2022-11-23T02:27:04.4572595Z Running tests...
2022-11-23T02:27:04.4573007Z ----------------------------------------------------------------------
2022-11-23T02:27:04.4573527Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-11-23T02:27:04.4574108Z   test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61136
2022-11-23T02:27:04.4574658Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61137
2022-11-23T02:27:04.4575267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:04.4575697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:04.4576272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:04.4577025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:04.4577608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:04.4578062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:04.4578634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:04.4579104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:04.4579542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp025ww1qj
2022-11-23T02:27:04.4580082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp025ww1qj/_remote_module_non_scriptable.py
2022-11-23T02:27:04.4580617Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm118ni81
2022-11-23T02:27:04.4581138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm118ni81/_remote_module_non_scriptable.py
2022-11-23T02:27:04.4581569Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:27:04.4582073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:04.4582471Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:27:04.4582856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:04.4583340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:04.4583839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:04.4584488Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:04.4585184Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:04.4585580Z ok (5.586s)
2022-11-23T02:27:04.4585733Z 
2022-11-23T02:27:04.4586004Z ----------------------------------------------------------------------
2022-11-23T02:27:04.4586321Z Ran 1 test in 5.586s
2022-11-23T02:27:04.4586485Z 
2022-11-23T02:27:04.4586578Z OK
2022-11-23T02:27:04.4586712Z 
2022-11-23T02:27:04.4586837Z Generating XML reports...
2022-11-23T02:27:04.4587466Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123022609.xml
2022-11-23T02:27:04.4588214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:04.4588663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:04.4589236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:04.4589687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:04.4590159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfnmol8v1
2022-11-23T02:27:04.4590704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfnmol8v1/_remote_module_non_scriptable.py
2022-11-23T02:27:04.4591136Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:27:04.4591315Z 
2022-11-23T02:27:04.4591424Z Running tests...
2022-11-23T02:27:04.4591833Z ----------------------------------------------------------------------
2022-11-23T02:27:04.4592371Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-11-23T02:27:04.4592930Z   test_allreduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61352
2022-11-23T02:27:04.4593572Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61353
2022-11-23T02:27:04.4594193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:04.4594724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:04.4595296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:04.4595770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:04.4596353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:04.4596779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:04.4597353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:04.4597818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:04.4598288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyj_dxe93
2022-11-23T02:27:04.4598806Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyj_dxe93/_remote_module_non_scriptable.py
2022-11-23T02:27:04.4599341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_27g9sft
2022-11-23T02:27:04.4599963Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_27g9sft/_remote_module_non_scriptable.py
2022-11-23T02:27:04.4600394Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:27:04.4600783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:04.4601172Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:27:04.4601579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:04.4602049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:04.4602545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:04.4603211Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:04.4603906Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:04.4604284Z ok (5.686s)
2022-11-23T02:27:04.4604433Z 
2022-11-23T02:27:04.4604700Z ----------------------------------------------------------------------
2022-11-23T02:27:04.4605030Z Ran 1 test in 5.686s
2022-11-23T02:27:04.4605192Z 
2022-11-23T02:27:04.4605265Z OK
2022-11-23T02:27:04.4605399Z 
2022-11-23T02:27:04.4605525Z Generating XML reports...
2022-11-23T02:27:04.4606170Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123022619.xml
2022-11-23T02:27:04.4606914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:04.4607347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:04.4607916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:04.4608396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:04.4608856Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7xlhxwy7
2022-11-23T02:27:04.4609387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7xlhxwy7/_remote_module_non_scriptable.py
2022-11-23T02:27:04.4609795Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:27:04.4609989Z 
2022-11-23T02:27:04.4610094Z Running tests...
2022-11-23T02:27:04.4610492Z ----------------------------------------------------------------------
2022-11-23T02:27:04.4611006Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-11-23T02:27:04.4611570Z   test_broadcast (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61568
2022-11-23T02:27:04.4612183Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61569
2022-11-23T02:27:04.4612796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:04.4613232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:04.4613806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:04.4614269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:04.4614844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:04.4615270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:04.4615835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:04.4616298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:04.4616938Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5u_pfm0e
2022-11-23T02:27:04.4617560Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5u_pfm0e/_remote_module_non_scriptable.py
2022-11-23T02:27:04.4618086Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgv9t5d70
2022-11-23T02:27:04.4618616Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgv9t5d70/_remote_module_non_scriptable.py
2022-11-23T02:27:04.4619017Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:27:04.4619419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:04.4619805Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:27:04.4620185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:04.4620667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:04.4621162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:04.4621837Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:04.4622510Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:04.4622904Z ok (5.685s)
2022-11-23T02:27:04.4623051Z 
2022-11-23T02:27:04.4623314Z ----------------------------------------------------------------------
2022-11-23T02:27:04.4623619Z Ran 1 test in 5.685s
2022-11-23T02:27:04.4623779Z 
2022-11-23T02:27:04.4623870Z OK
2022-11-23T02:27:04.4624000Z 
2022-11-23T02:27:04.4624123Z Generating XML reports...
2022-11-23T02:27:04.4624758Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123022629.xml
2022-11-23T02:27:04.4625484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:04.4625933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:04.4626503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:04.4626955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:04.4627420Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyesm8an6
2022-11-23T02:27:04.4627960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyesm8an6/_remote_module_non_scriptable.py
2022-11-23T02:27:04.4628382Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:27:04.4628577Z 
2022-11-23T02:27:04.4628667Z Running tests...
2022-11-23T02:27:04.4629077Z ----------------------------------------------------------------------
2022-11-23T02:27:04.4629684Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-11-23T02:27:04.4630240Z   test_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61784
2022-11-23T02:27:04.4630772Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61785
2022-11-23T02:27:04.4631379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:04.4631825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:04.4632376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:04.4632839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:04.4633413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:04.4633851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:04.4634408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:04.4634947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:04.4635407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmjbcmcd7
2022-11-23T02:27:04.4635924Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmjbcmcd7/_remote_module_non_scriptable.py
2022-11-23T02:27:04.4636451Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy96fjw_i
2022-11-23T02:27:04.4636982Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy96fjw_i/_remote_module_non_scriptable.py
2022-11-23T02:27:04.4637404Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:27:04.4637791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:04.4638180Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:27:04.4638582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:04.4639052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:04.4639542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:04.4640203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:04.4640888Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:04.4641262Z ok (5.484s)
2022-11-23T02:27:04.4641408Z 
2022-11-23T02:27:04.4641671Z ----------------------------------------------------------------------
2022-11-23T02:27:04.4641996Z Ran 1 test in 5.484s
2022-11-23T02:27:04.4642156Z 
2022-11-23T02:27:04.4642230Z OK
2022-11-23T02:27:04.4642361Z 
2022-11-23T02:27:04.4642482Z Generating XML reports...
2022-11-23T02:27:04.4643127Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123022638.xml
2022-11-23T02:27:04.4643864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:04.4644291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:04.4644863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:04.4645328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:04.4645791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0qlblnqz
2022-11-23T02:27:04.4646307Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0qlblnqz/_remote_module_non_scriptable.py
2022-11-23T02:27:04.4646735Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:27:04.4646929Z 
2022-11-23T02:27:04.4647101Z Running tests...
2022-11-23T02:27:04.4647497Z ----------------------------------------------------------------------
2022-11-23T02:27:04.4648034Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-11-23T02:27:04.4648601Z   test_reduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62000
2022-11-23T02:27:04.4649133Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62001
2022-11-23T02:27:04.4649719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:04.4650164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:04.4650731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:04.4651179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:04.4651750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:04.4652264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:04.4652831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:04.4653272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:04.4653743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptvgoromt
2022-11-23T02:27:04.4654336Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptvgoromt/_remote_module_non_scriptable.py
2022-11-23T02:27:04.4654854Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpptvcvb2_
2022-11-23T02:27:04.4655384Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpptvcvb2_/_remote_module_non_scriptable.py
2022-11-23T02:27:04.4655811Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:27:04.4656215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:04.4656586Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:27:04.4657221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:04.4657703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:04.4658182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:04.4658843Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:04.4659528Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:04.4659920Z ok (5.487s)
2022-11-23T02:27:04.4660065Z 
2022-11-23T02:27:04.4660318Z ----------------------------------------------------------------------
2022-11-23T02:27:04.4660703Z Ran 1 test in 5.487s
2022-11-23T02:27:04.4660870Z 
2022-11-23T02:27:04.4660961Z OK
2022-11-23T02:27:04.4661094Z 
2022-11-23T02:27:04.4661201Z Generating XML reports...
2022-11-23T02:27:04.4661846Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123022648.xml
2022-11-23T02:27:04.4662582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:04.4663028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:04.4663578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:04.4664040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:04.4664504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt4_7ai7v
2022-11-23T02:27:04.4665129Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt4_7ai7v/_remote_module_non_scriptable.py
2022-11-23T02:27:04.4665554Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:27:04.4665752Z 
2022-11-23T02:27:04.4665863Z Running tests...
2022-11-23T02:27:04.4666273Z ----------------------------------------------------------------------
2022-11-23T02:27:04.4666794Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-11-23T02:27:04.4667371Z   test_scatter (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62216
2022-11-23T02:27:04.4667903Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62217
2022-11-23T02:27:04.4668507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:04.4668941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:04.4669515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:04.4670063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:04.4670629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:04.4671071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:04.4671638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:04.4672100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:04.4672547Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp16o9fwve
2022-11-23T02:27:04.4673080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp16o9fwve/_remote_module_non_scriptable.py
2022-11-23T02:27:04.4673610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmhrz7iz3
2022-11-23T02:27:04.4674144Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmhrz7iz3/_remote_module_non_scriptable.py
2022-11-23T02:27:04.4674552Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:27:04.4674955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:04.4675346Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:27:04.4675729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:04.4676209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:04.4676867Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:04.4677404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:04.4678034Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:04.4678424Z ok (5.584s)
2022-11-23T02:27:04.4678570Z 
2022-11-23T02:27:04.4678843Z ----------------------------------------------------------------------
2022-11-23T02:27:04.4679147Z Ran 1 test in 5.585s
2022-11-23T02:27:04.4679304Z 
2022-11-23T02:27:04.4679396Z OK
2022-11-23T02:27:04.4679529Z 
2022-11-23T02:27:04.4679650Z Generating XML reports...
2022-11-23T02:27:04.4680290Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123022657.xml
2022-11-23T02:27:04.4680662Z 
2022-11-23T02:27:04.4681215Z ##[endgroup]
2022-11-23T02:27:04.4681793Z FINISHED PRINTING LOG FILE of distributed/test_c10d_spawn_gloo (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_gloo_z95s6kkl)
2022-11-23T02:27:04.4682134Z 
2022-11-23T02:27:04.8419618Z 
2022-11-23T02:27:04.8420054Z real	1m41.421s
2022-11-23T02:27:04.8420686Z user	2m38.970s
2022-11-23T02:27:04.8420942Z sys	2m9.277s
2022-11-23T02:27:04.8421515Z + python test/run_test.py --verbose -i distributed/test_c10d_spawn_nccl
2022-11-23T02:27:07.2719229Z Ignoring disabled issues:  []
2022-11-23T02:27:07.3239536Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T02:27:07.3240563Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T02:27:07.3240922Z Selected tests:
2022-11-23T02:27:07.3241177Z  distributed/test_c10d_spawn_nccl
2022-11-23T02:27:07.3271464Z Prioritized test from test file changes.
2022-11-23T02:27:07.3271862Z reordering tests for PR:
2022-11-23T02:27:07.3272135Z prioritized: []
2022-11-23T02:27:07.3272686Z the rest: ['distributed/test_c10d_spawn_nccl']
2022-11-23T02:27:07.3272786Z 
2022-11-23T02:27:07.3273449Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T02:27:07.3274423Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T02:27:07.3278843Z parallel (file granularity) tests:
2022-11-23T02:27:07.3279107Z  
2022-11-23T02:27:07.3279449Z serial (file granularity) tests:
2022-11-23T02:27:07.3279735Z  distributed/test_c10d_spawn_nccl
2022-11-23T02:27:09.5669861Z Ignoring disabled issues:  []
2022-11-23T02:27:09.6348978Z Ignoring disabled issues:  []
2022-11-23T02:27:10.0201959Z Running distributed/test_c10d_spawn_nccl ... [2022-11-23 02:27:10.019542]
2022-11-23T02:27:10.0203072Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_spawn_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:27:10.020006]
2022-11-23T02:28:40.7178044Z 
2022-11-23T02:28:40.7179668Z Expand the folded group to see the log file of distributed/test_c10d_spawn_nccl
2022-11-23T02:28:40.7180675Z ##[group]PRINTING LOG FILE of distributed/test_c10d_spawn_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_nccl_d9jysm_1)
2022-11-23T02:28:40.7181294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjf6azrhp
2022-11-23T02:28:40.7181838Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjf6azrhp/_remote_module_non_scriptable.py
2022-11-23T02:28:40.7182294Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:28:40.7182634Z <unittest.suite.TestSuite tests=[]>
2022-11-23T02:28:40.7182954Z <unittest.suite.TestSuite tests=[]>
2022-11-23T02:28:40.7184246Z <unittest.suite.TestSuite tests=[<__main__.TestDistributedNNFunctionsNccl testMethod=test_all_gather>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_gather_base>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce_scatter>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce_scatter_non_contiguous>]>
2022-11-23T02:28:40.7186030Z test_all_gather (__main__.TestDistributedNNFunctionsNccl)
2022-11-23T02:28:40.7186432Z test_all_gather_base (__main__.TestDistributedNNFunctionsNccl)
2022-11-23T02:28:40.7186859Z test_all_to_all (__main__.TestDistributedNNFunctionsNccl)
2022-11-23T02:28:40.7187264Z test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl)
2022-11-23T02:28:40.7187676Z test_allreduce (__main__.TestDistributedNNFunctionsNccl)
2022-11-23T02:28:40.7188055Z test_broadcast (__main__.TestDistributedNNFunctionsNccl)
2022-11-23T02:28:40.7188448Z test_reduce (__main__.TestDistributedNNFunctionsNccl)
2022-11-23T02:28:40.7189180Z test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl)
2022-11-23T02:28:40.7196057Z test_reduce_scatter_non_contiguous (__main__.TestDistributedNNFunctionsNccl)
2022-11-23T02:28:40.7196851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:40.7197330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:40.7197914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:40.7198391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:40.7198851Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplwa5i537
2022-11-23T02:28:40.7199400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplwa5i537/_remote_module_non_scriptable.py
2022-11-23T02:28:40.7199825Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:28:40.7200032Z 
2022-11-23T02:28:40.7200133Z Running tests...
2022-11-23T02:28:40.7200548Z ----------------------------------------------------------------------
2022-11-23T02:28:40.7201300Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-11-23T02:28:40.7201886Z   test_all_gather (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62713
2022-11-23T02:28:40.7202411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62714
2022-11-23T02:28:40.7203023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:40.7203481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:40.7204040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:40.7204539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:40.7205129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:40.7205646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:40.7206225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:40.7206674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:40.7207147Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7nmkha3u
2022-11-23T02:28:40.7207698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7nmkha3u/_remote_module_non_scriptable.py
2022-11-23T02:28:40.7208241Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp923nizh8
2022-11-23T02:28:40.7208756Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp923nizh8/_remote_module_non_scriptable.py
2022-11-23T02:28:40.7209219Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:28:40.7209543Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:28:40.7209936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:40.7210446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:40.7210933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:40.7211429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:40.7212081Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:40.7212786Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:40.7213188Z ok (5.488s)
2022-11-23T02:28:40.7213348Z 
2022-11-23T02:28:40.7213697Z ----------------------------------------------------------------------
2022-11-23T02:28:40.7214021Z Ran 1 test in 5.489s
2022-11-23T02:28:40.7214193Z 
2022-11-23T02:28:40.7214290Z OK
2022-11-23T02:28:40.7214427Z 
2022-11-23T02:28:40.7214554Z Generating XML reports...
2022-11-23T02:28:40.7215184Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022717.xml
2022-11-23T02:28:40.7215924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:40.7216377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:40.7217268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:40.7217729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:40.7218199Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2jl54eke
2022-11-23T02:28:40.7218740Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2jl54eke/_remote_module_non_scriptable.py
2022-11-23T02:28:40.7219255Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:28:40.7219451Z 
2022-11-23T02:28:40.7219561Z Running tests...
2022-11-23T02:28:40.7219973Z ----------------------------------------------------------------------
2022-11-23T02:28:40.7220514Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-11-23T02:28:40.7221079Z   test_all_gather_base (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62932
2022-11-23T02:28:40.7221629Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62933
2022-11-23T02:28:40.7222242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:40.7222677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:40.7223322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:40.7223799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:40.7224377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:40.7224804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:40.7225369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:40.7225832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:40.7226273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3286w1rt
2022-11-23T02:28:40.7226811Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3286w1rt/_remote_module_non_scriptable.py
2022-11-23T02:28:40.7227343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjkqok55p
2022-11-23T02:28:40.7227882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjkqok55p/_remote_module_non_scriptable.py
2022-11-23T02:28:40.7243721Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:28:40.7244156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:40.7244676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:40.7245105Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:28:40.7245513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:40.7246038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:40.7246772Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:40.7247645Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:40.7248635Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:28:40.7249291Z   warnings.warn(
2022-11-23T02:28:40.7250092Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:28:40.7250671Z   warnings.warn(
2022-11-23T02:28:40.7251473Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2923: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-11-23T02:28:40.7252066Z   warnings.warn(
2022-11-23T02:28:40.7252877Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2923: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-11-23T02:28:40.7253535Z   warnings.warn(
2022-11-23T02:28:40.7253765Z ok (5.488s)
2022-11-23T02:28:40.7253925Z 
2022-11-23T02:28:40.7254217Z ----------------------------------------------------------------------
2022-11-23T02:28:40.7254564Z Ran 1 test in 5.489s
2022-11-23T02:28:40.7254731Z 
2022-11-23T02:28:40.7254823Z OK
2022-11-23T02:28:40.7254942Z 
2022-11-23T02:28:40.7255072Z Generating XML reports...
2022-11-23T02:28:40.7255751Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022726.xml
2022-11-23T02:28:40.7256540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:40.7257291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:40.7257907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:40.7258413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:40.7258904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcwdxbj2f
2022-11-23T02:28:40.7259453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcwdxbj2f/_remote_module_non_scriptable.py
2022-11-23T02:28:40.7259903Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:28:40.7260108Z 
2022-11-23T02:28:40.7260377Z Running tests...
2022-11-23T02:28:40.7260766Z ----------------------------------------------------------------------
2022-11-23T02:28:40.7261294Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-11-23T02:28:40.7261858Z   test_all_to_all (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63151
2022-11-23T02:28:40.7262674Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63152
2022-11-23T02:28:40.7263271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:40.7263723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:40.7264298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:40.7264772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:40.7265331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:40.7265779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:40.7266446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:40.7267066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:40.7267513Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpei_7bv1t
2022-11-23T02:28:40.7268040Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpei_7bv1t/_remote_module_non_scriptable.py
2022-11-23T02:28:40.7268559Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1fxdbvj7
2022-11-23T02:28:40.7269061Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1fxdbvj7/_remote_module_non_scriptable.py
2022-11-23T02:28:40.7269475Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:28:40.7269876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:40.7270333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:40.7270733Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:28:40.7271126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:40.7271669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:40.7272294Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:40.7272957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:40.7273337Z ok (5.682s)
2022-11-23T02:28:40.7273480Z 
2022-11-23T02:28:40.7273741Z ----------------------------------------------------------------------
2022-11-23T02:28:40.7274041Z Ran 1 test in 5.683s
2022-11-23T02:28:40.7274198Z 
2022-11-23T02:28:40.7274289Z OK
2022-11-23T02:28:40.7274420Z 
2022-11-23T02:28:40.7274541Z Generating XML reports...
2022-11-23T02:28:40.7275150Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022736.xml
2022-11-23T02:28:40.7275873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:40.7276309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:40.7276861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:40.7277298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:40.7277747Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptnwklox6
2022-11-23T02:28:40.7278273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptnwklox6/_remote_module_non_scriptable.py
2022-11-23T02:28:40.7278669Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:28:40.7278860Z 
2022-11-23T02:28:40.7278964Z Running tests...
2022-11-23T02:28:40.7279360Z ----------------------------------------------------------------------
2022-11-23T02:28:40.7279883Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-11-23T02:28:40.7280425Z   test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63372
2022-11-23T02:28:40.7280957Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63373
2022-11-23T02:28:40.7281540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:40.7282135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:40.7282712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:40.7283178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:40.7283818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:40.7284252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:40.7284825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:40.7285288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:40.7285732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptuw12u_m
2022-11-23T02:28:40.7286273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptuw12u_m/_remote_module_non_scriptable.py
2022-11-23T02:28:40.7286805Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu_695kyc
2022-11-23T02:28:40.7287338Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu_695kyc/_remote_module_non_scriptable.py
2022-11-23T02:28:40.7287742Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:28:40.7288159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:40.7288872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:40.7289262Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:28:40.7289633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:40.7290290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:40.7290949Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:40.7291622Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:40.7292016Z ok (5.687s)
2022-11-23T02:28:40.7292164Z 
2022-11-23T02:28:40.7292439Z ----------------------------------------------------------------------
2022-11-23T02:28:40.7292769Z Ran 1 test in 5.688s
2022-11-23T02:28:40.7292915Z 
2022-11-23T02:28:40.7293011Z OK
2022-11-23T02:28:40.7293146Z 
2022-11-23T02:28:40.7293269Z Generating XML reports...
2022-11-23T02:28:40.7294021Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022746.xml
2022-11-23T02:28:40.7294747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:40.7295203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:40.7295781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:40.7296250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:40.7296883Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe1msguc8
2022-11-23T02:28:40.7297451Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe1msguc8/_remote_module_non_scriptable.py
2022-11-23T02:28:40.7297881Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:28:40.7298078Z 
2022-11-23T02:28:40.7298168Z Running tests...
2022-11-23T02:28:40.7298578Z ----------------------------------------------------------------------
2022-11-23T02:28:40.7299116Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-11-23T02:28:40.7299691Z   test_allreduce (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63593
2022-11-23T02:28:40.7300215Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63594
2022-11-23T02:28:40.7300966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:40.7301402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:40.7302027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:40.7302494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:40.7303341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:40.7303787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:40.7304337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:40.7304800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:40.7305262Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2c2l39hv
2022-11-23T02:28:40.7305807Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2c2l39hv/_remote_module_non_scriptable.py
2022-11-23T02:28:40.7306470Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqqbwc_vf
2022-11-23T02:28:40.7306985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqqbwc_vf/_remote_module_non_scriptable.py
2022-11-23T02:28:40.7307688Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:28:40.7307989Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:28:40.7308387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:40.7308876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:40.7309362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:40.7309824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:40.7310496Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:40.7311188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:40.7311567Z ok (5.483s)
2022-11-23T02:28:40.7311716Z 
2022-11-23T02:28:40.7311985Z ----------------------------------------------------------------------
2022-11-23T02:28:40.7312315Z Ran 1 test in 5.484s
2022-11-23T02:28:40.7312477Z 
2022-11-23T02:28:40.7312570Z OK
2022-11-23T02:28:40.7312684Z 
2022-11-23T02:28:40.7312808Z Generating XML reports...
2022-11-23T02:28:40.7313455Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022755.xml
2022-11-23T02:28:40.7314198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:40.7314629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:40.7315208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:40.7315683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:40.7316309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfsfn9qm6
2022-11-23T02:28:40.7316988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfsfn9qm6/_remote_module_non_scriptable.py
2022-11-23T02:28:40.7317419Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:28:40.7317615Z 
2022-11-23T02:28:40.7317727Z Running tests...
2022-11-23T02:28:40.7318118Z ----------------------------------------------------------------------
2022-11-23T02:28:40.7318656Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-11-23T02:28:40.7319227Z   test_broadcast (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63812
2022-11-23T02:28:40.7319934Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63813
2022-11-23T02:28:40.7320567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:40.7321013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:40.7321569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:40.7322025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:40.7322565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:40.7322990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:40.7323724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:40.7324170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:40.7324642Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbjl4o3rb
2022-11-23T02:28:40.7325182Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbjl4o3rb/_remote_module_non_scriptable.py
2022-11-23T02:28:40.7325785Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3lnoffsl
2022-11-23T02:28:40.7326301Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3lnoffsl/_remote_module_non_scriptable.py
2022-11-23T02:28:40.7326728Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:28:40.7327135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:40.7327752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:40.7328140Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:28:40.7328530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:40.7329002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:40.7329629Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:40.7330298Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:40.7330675Z ok (5.582s)
2022-11-23T02:28:40.7330819Z 
2022-11-23T02:28:40.7331060Z ----------------------------------------------------------------------
2022-11-23T02:28:40.7331375Z Ran 1 test in 5.583s
2022-11-23T02:28:40.7331530Z 
2022-11-23T02:28:40.7331619Z OK
2022-11-23T02:28:40.7331751Z 
2022-11-23T02:28:40.7331874Z Generating XML reports...
2022-11-23T02:28:40.7332478Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022805.xml
2022-11-23T02:28:40.7333195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:40.7333648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:40.7334187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:40.7334648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:40.7335108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp51flr778
2022-11-23T02:28:40.7335640Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp51flr778/_remote_module_non_scriptable.py
2022-11-23T02:28:40.7336037Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:28:40.7336230Z 
2022-11-23T02:28:40.7336341Z Running tests...
2022-11-23T02:28:40.7337104Z ----------------------------------------------------------------------
2022-11-23T02:28:40.7337659Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-11-23T02:28:40.7338304Z   test_reduce (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64031
2022-11-23T02:28:40.7338872Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64032
2022-11-23T02:28:40.7339494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:40.7339931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:40.7340673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:40.7341139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:40.7341707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:40.7342119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:40.7342729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:40.7343216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:40.7343736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8kuv6p52
2022-11-23T02:28:40.7344267Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8kuv6p52/_remote_module_non_scriptable.py
2022-11-23T02:28:40.7344793Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp33lb1_ea
2022-11-23T02:28:40.7345318Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp33lb1_ea/_remote_module_non_scriptable.py
2022-11-23T02:28:40.7345715Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:28:40.7346116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:40.7346597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:40.7346976Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:28:40.7347380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:40.7347866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:40.7348521Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:40.7349369Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:40.7349778Z ok (5.583s)
2022-11-23T02:28:40.7349932Z 
2022-11-23T02:28:40.7350204Z ----------------------------------------------------------------------
2022-11-23T02:28:40.7350520Z Ran 1 test in 5.584s
2022-11-23T02:28:40.7350688Z 
2022-11-23T02:28:40.7350786Z OK
2022-11-23T02:28:40.7350923Z 
2022-11-23T02:28:40.7351051Z Generating XML reports...
2022-11-23T02:28:40.7351715Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022815.xml
2022-11-23T02:28:40.7352606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:40.7353060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:40.7353627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:40.7354095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:40.7354708Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsilyqgus
2022-11-23T02:28:40.7355263Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsilyqgus/_remote_module_non_scriptable.py
2022-11-23T02:28:40.7355705Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:28:40.7355907Z 
2022-11-23T02:28:40.7355996Z Running tests...
2022-11-23T02:28:40.7356470Z ----------------------------------------------------------------------
2022-11-23T02:28:40.7357033Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-11-23T02:28:40.7357780Z   test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64250
2022-11-23T02:28:40.7358297Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64251
2022-11-23T02:28:40.7358895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:40.7359338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:40.7359877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:40.7360337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:40.7360908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:40.7361347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:40.7361947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:40.7362404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:40.7363044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgan1fzq4
2022-11-23T02:28:40.7363579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgan1fzq4/_remote_module_non_scriptable.py
2022-11-23T02:28:40.7364119Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8d6ygodj
2022-11-23T02:28:40.7364670Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8d6ygodj/_remote_module_non_scriptable.py
2022-11-23T02:28:40.7365105Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:28:40.7365503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:40.7366003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:40.7366418Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:28:40.7366805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:40.7367306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:40.7368230Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:40.7368902Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:40.7369266Z ok (5.485s)
2022-11-23T02:28:40.7369420Z 
2022-11-23T02:28:40.7369686Z ----------------------------------------------------------------------
2022-11-23T02:28:40.7370018Z Ran 1 test in 5.487s
2022-11-23T02:28:40.7370182Z 
2022-11-23T02:28:40.7370280Z OK
2022-11-23T02:28:40.7370392Z 
2022-11-23T02:28:40.7370521Z Generating XML reports...
2022-11-23T02:28:40.7371156Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022824.xml
2022-11-23T02:28:40.7371884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:40.7372304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:40.7372870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:40.7373341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:40.7373805Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3403okhq
2022-11-23T02:28:40.7374373Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3403okhq/_remote_module_non_scriptable.py
2022-11-23T02:28:40.7374798Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:28:40.7375000Z 
2022-11-23T02:28:40.7375113Z Running tests...
2022-11-23T02:28:40.7375496Z ----------------------------------------------------------------------
2022-11-23T02:28:40.7376032Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-11-23T02:28:40.7376614Z   test_reduce_scatter_non_contiguous (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64469
2022-11-23T02:28:40.7377688Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64470
2022-11-23T02:28:40.7378289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:40.7378747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:40.7379341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:40.7379802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:40.7380489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:40.7380947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:40.7381530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:40.7381985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:40.7382459Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9lxszo41
2022-11-23T02:28:40.7383007Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9lxszo41/_remote_module_non_scriptable.py
2022-11-23T02:28:40.7383553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdoxmm0sh
2022-11-23T02:28:40.7384081Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdoxmm0sh/_remote_module_non_scriptable.py
2022-11-23T02:28:40.7384527Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:28:40.7384941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:40.7385420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:40.7385835Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:28:40.7386246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:40.7386741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:40.7387391Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:40.7388086Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:40.7388496Z ok (5.585s)
2022-11-23T02:28:40.7388655Z 
2022-11-23T02:28:40.7388907Z ----------------------------------------------------------------------
2022-11-23T02:28:40.7389240Z Ran 1 test in 5.586s
2022-11-23T02:28:40.7389402Z 
2022-11-23T02:28:40.7389495Z OK
2022-11-23T02:28:40.7389633Z 
2022-11-23T02:28:40.7389922Z Generating XML reports...
2022-11-23T02:28:40.7390718Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022834.xml
2022-11-23T02:28:40.7391111Z 
2022-11-23T02:28:40.7391581Z ##[endgroup]
2022-11-23T02:28:40.7392177Z FINISHED PRINTING LOG FILE of distributed/test_c10d_spawn_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_nccl_d9jysm_1)
2022-11-23T02:28:40.7392523Z 
2022-11-23T02:28:41.0488857Z 
2022-11-23T02:28:41.0489407Z real	1m36.207s
2022-11-23T02:28:41.0489690Z user	2m43.299s
2022-11-23T02:28:41.0489919Z sys	2m2.598s
2022-11-23T02:28:41.0490543Z + python test/run_test.py --verbose -i distributed/test_store
2022-11-23T02:28:43.4427173Z Ignoring disabled issues:  []
2022-11-23T02:28:43.4940494Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T02:28:43.4941127Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T02:28:43.4941491Z Selected tests:
2022-11-23T02:28:43.4941729Z  distributed/test_store
2022-11-23T02:28:43.4966867Z Prioritized test from test file changes.
2022-11-23T02:28:43.4967301Z reordering tests for PR:
2022-11-23T02:28:43.4967788Z prioritized: []
2022-11-23T02:28:43.4968232Z the rest: ['distributed/test_store']
2022-11-23T02:28:43.4968424Z 
2022-11-23T02:28:43.4968967Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T02:28:43.4969943Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T02:28:43.4975468Z parallel (file granularity) tests:
2022-11-23T02:28:43.4975759Z  
2022-11-23T02:28:43.4976028Z serial (file granularity) tests:
2022-11-23T02:28:43.4976329Z  distributed/test_store
2022-11-23T02:28:45.7580866Z Ignoring disabled issues:  []
2022-11-23T02:28:45.7701820Z Ignoring disabled issues:  []
2022-11-23T02:28:46.1540919Z Running distributed/test_store ... [2022-11-23 02:28:46.153512]
2022-11-23T02:28:46.1543182Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_store.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:28:46.153993]
2022-11-23T02:30:58.0192079Z 
2022-11-23T02:30:58.0192672Z Expand the folded group to see the log file of distributed/test_store
2022-11-23T02:30:58.0193670Z ##[group]PRINTING LOG FILE of distributed/test_store (/var/lib/jenkins/workspace/test/test-reports/distributed-test_store_wcuh6d55)
2022-11-23T02:30:58.0200317Z <unittest.suite.TestSuite tests=[<__main__.FileStoreTest testMethod=test_compare_set>, <__main__.FileStoreTest testMethod=test_init_pg_and_rpc_with_same_file>, <__main__.FileStoreTest testMethod=test_refcount>, <__main__.FileStoreTest testMethod=test_set_get>]>
2022-11-23T02:30:58.0201024Z test_compare_set (__main__.FileStoreTest)
2022-11-23T02:30:58.0201500Z test_init_pg_and_rpc_with_same_file (__main__.FileStoreTest)
2022-11-23T02:30:58.0202028Z test_refcount (__main__.FileStoreTest)
2022-11-23T02:30:58.0202339Z test_set_get (__main__.FileStoreTest)
2022-11-23T02:30:58.0203099Z <unittest.suite.TestSuite tests=[<__main__.HashStoreTest testMethod=test_compare_set>, <__main__.HashStoreTest testMethod=test_set_get>]>
2022-11-23T02:30:58.0203799Z test_compare_set (__main__.HashStoreTest)
2022-11-23T02:30:58.0204126Z test_set_get (__main__.HashStoreTest)
2022-11-23T02:30:58.0204639Z <unittest.suite.TestSuite tests=[<__main__.PrefixFileStoreTest testMethod=test_compare_set>, <__main__.PrefixFileStoreTest testMethod=test_set_get>]>
2022-11-23T02:30:58.0205129Z test_compare_set (__main__.PrefixFileStoreTest)
2022-11-23T02:30:58.0205475Z test_set_get (__main__.PrefixFileStoreTest)
2022-11-23T02:30:58.0205907Z <unittest.suite.TestSuite tests=[<__main__.PrefixStoreTest testMethod=test_get_underlying_store>]>
2022-11-23T02:30:58.0206526Z test_get_underlying_store (__main__.PrefixStoreTest)
2022-11-23T02:30:58.0207159Z <unittest.suite.TestSuite tests=[<__main__.PrefixTCPStoreTest testMethod=test_compare_set>, <__main__.PrefixTCPStoreTest testMethod=test_set_get>]>
2022-11-23T02:30:58.0208727Z test_compare_set (__main__.PrefixTCPStoreTest)
2022-11-23T02:30:58.0209355Z test_set_get (__main__.PrefixTCPStoreTest)
2022-11-23T02:30:58.0210095Z <unittest.suite.TestSuite tests=[<__main__.PythonStoreTest testMethod=test_set_get>]>
2022-11-23T02:30:58.0210766Z test_set_get (__main__.PythonStoreTest)
2022-11-23T02:30:58.0211759Z <unittest.suite.TestSuite tests=[<__main__.RendezvousEnvTest testMethod=test_nominal>]>
2022-11-23T02:30:58.0212459Z test_nominal (__main__.RendezvousEnvTest)
2022-11-23T02:30:58.0213245Z <unittest.suite.TestSuite tests=[<__main__.RendezvousFileTest testMethod=test_common_errors>, <__main__.RendezvousFileTest testMethod=test_nominal>]>
2022-11-23T02:30:58.0214139Z test_common_errors (__main__.RendezvousFileTest)
2022-11-23T02:30:58.0214692Z test_nominal (__main__.RendezvousFileTest)
2022-11-23T02:30:58.0215742Z <unittest.suite.TestSuite tests=[<__main__.RendezvousTCPTest testMethod=test_common_errors>, <__main__.RendezvousTCPTest testMethod=test_dns_timeout>, <__main__.RendezvousTCPTest testMethod=test_nominal>, <__main__.RendezvousTCPTest testMethod=test_tcp_store_timeout_set>]>
2022-11-23T02:30:58.0217128Z test_common_errors (__main__.RendezvousTCPTest)
2022-11-23T02:30:58.0217702Z test_dns_timeout (__main__.RendezvousTCPTest)
2022-11-23T02:30:58.0218039Z test_nominal (__main__.RendezvousTCPTest)
2022-11-23T02:30:58.0218374Z test_tcp_store_timeout_set (__main__.RendezvousTCPTest)
2022-11-23T02:30:58.0219076Z <unittest.suite.TestSuite tests=[<__main__.RendezvousTest testMethod=test_unknown_handler>, <__main__.RendezvousTest testMethod=test_url_with_node_params>]>
2022-11-23T02:30:58.0219752Z test_unknown_handler (__main__.RendezvousTest)
2022-11-23T02:30:58.0220380Z test_url_with_node_params (__main__.RendezvousTest)
2022-11-23T02:30:58.0222043Z <unittest.suite.TestSuite tests=[<__main__.TCPStoreTest testMethod=test_address_already_in_use>, <__main__.TCPStoreTest testMethod=test_compare_set>, <__main__.TCPStoreTest testMethod=test_init_pg_and_rpc_with_same_socket>, <__main__.TCPStoreTest testMethod=test_multi_worker_with_fixed_world_size>, <__main__.TCPStoreTest testMethod=test_multi_worker_with_nonfixed_world_size>, <__main__.TCPStoreTest testMethod=test_multitenancy>, <__main__.TCPStoreTest testMethod=test_numkeys_delkeys>, <__main__.TCPStoreTest testMethod=test_set_get>]>
2022-11-23T02:30:58.0223696Z test_address_already_in_use (__main__.TCPStoreTest)
2022-11-23T02:30:58.0224025Z test_compare_set (__main__.TCPStoreTest)
2022-11-23T02:30:58.0224376Z test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest)
2022-11-23T02:30:58.0224755Z test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest)
2022-11-23T02:30:58.0225118Z test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest)
2022-11-23T02:30:58.0225703Z test_multitenancy (__main__.TCPStoreTest)
2022-11-23T02:30:58.0226032Z test_numkeys_delkeys (__main__.TCPStoreTest)
2022-11-23T02:30:58.0226326Z test_set_get (__main__.TCPStoreTest)
2022-11-23T02:30:58.0227023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:58.0227484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:58.0228070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:58.0228533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:58.0228771Z 
2022-11-23T02:30:58.0228882Z Running tests...
2022-11-23T02:30:58.0229302Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0229810Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:30:58.0230288Z   test_compare_set (__main__.FileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:30:58.0230631Z ok (1.734s)
2022-11-23T02:30:58.0230781Z 
2022-11-23T02:30:58.0231051Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0231363Z Ran 1 test in 1.735s
2022-11-23T02:30:58.0231525Z 
2022-11-23T02:30:58.0231622Z OK
2022-11-23T02:30:58.0231757Z 
2022-11-23T02:30:58.0231882Z Generating XML reports...
2022-11-23T02:30:58.0232427Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123022850.xml
2022-11-23T02:30:58.0233238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:58.0233720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:58.0234306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:58.0234765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:58.0234998Z 
2022-11-23T02:30:58.0235108Z Running tests...
2022-11-23T02:30:58.0235522Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0236029Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:30:58.0236519Z   test_init_pg_and_rpc_with_same_file (__main__.FileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:30:58.0237074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:58.0237752Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:30:58.0238194Z ok (1.773s)
2022-11-23T02:30:58.0238344Z 
2022-11-23T02:30:58.0238614Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0238950Z Ran 1 test in 1.773s
2022-11-23T02:30:58.0239117Z 
2022-11-23T02:30:58.0239191Z OK
2022-11-23T02:30:58.0239331Z 
2022-11-23T02:30:58.0239461Z Generating XML reports...
2022-11-23T02:30:58.0240033Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123022854.xml
2022-11-23T02:30:58.0240724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:58.0241168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:58.0241760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:58.0242243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:58.0242487Z 
2022-11-23T02:30:58.0242577Z Running tests...
2022-11-23T02:30:58.0242998Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0243540Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:30:58.0244012Z   test_refcount (__main__.FileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:30:58.0244336Z ok (1.728s)
2022-11-23T02:30:58.0244492Z 
2022-11-23T02:30:58.0244766Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0245109Z Ran 1 test in 1.729s
2022-11-23T02:30:58.0245278Z 
2022-11-23T02:30:58.0245351Z OK
2022-11-23T02:30:58.0245491Z 
2022-11-23T02:30:58.0245622Z Generating XML reports...
2022-11-23T02:30:58.0246190Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123022858.xml
2022-11-23T02:30:58.0246884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:58.0247333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:58.0247931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:58.0248420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:58.0248664Z 
2022-11-23T02:30:58.0248755Z Running tests...
2022-11-23T02:30:58.0249177Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0249709Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:30:58.0250185Z   test_set_get (__main__.FileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:30:58.0250501Z ok (1.739s)
2022-11-23T02:30:58.0250660Z 
2022-11-23T02:30:58.0250995Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0251340Z Ran 1 test in 1.739s
2022-11-23T02:30:58.0251512Z 
2022-11-23T02:30:58.0251586Z OK
2022-11-23T02:30:58.0251728Z 
2022-11-23T02:30:58.0251860Z Generating XML reports...
2022-11-23T02:30:58.0252428Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123022902.xml
2022-11-23T02:30:58.0253118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:58.0253562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:58.0254148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:58.0254636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:58.0254873Z 
2022-11-23T02:30:58.0254965Z Running tests...
2022-11-23T02:30:58.0255385Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0255922Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:30:58.0256465Z   test_compare_set (__main__.HashStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:30:58.0257097Z ok (1.715s)
2022-11-23T02:30:58.0257256Z 
2022-11-23T02:30:58.0257539Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0257877Z Ran 1 test in 1.715s
2022-11-23T02:30:58.0258046Z 
2022-11-23T02:30:58.0258120Z OK
2022-11-23T02:30:58.0258260Z 
2022-11-23T02:30:58.0258391Z Generating XML reports...
2022-11-23T02:30:58.0258960Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20221123022906.xml
2022-11-23T02:30:58.0259626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:58.0260101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:58.0260693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:58.0261181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:58.0261418Z 
2022-11-23T02:30:58.0261507Z Running tests...
2022-11-23T02:30:58.0261925Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0262460Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:30:58.0262903Z   test_set_get (__main__.HashStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:30:58.0263247Z ok (1.727s)
2022-11-23T02:30:58.0263401Z 
2022-11-23T02:30:58.0263673Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0264011Z Ran 1 test in 1.727s
2022-11-23T02:30:58.0264158Z 
2022-11-23T02:30:58.0264256Z OK
2022-11-23T02:30:58.0264393Z 
2022-11-23T02:30:58.0264528Z Generating XML reports...
2022-11-23T02:30:58.0265100Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20221123022910.xml
2022-11-23T02:30:58.0265770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:58.0266235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:58.0266825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:58.0267308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:58.0267547Z 
2022-11-23T02:30:58.0267638Z Running tests...
2022-11-23T02:30:58.0268065Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0268598Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:30:58.0269066Z   test_compare_set (__main__.PrefixFileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:30:58.0269434Z ok (1.714s)
2022-11-23T02:30:58.0269688Z 
2022-11-23T02:30:58.0269973Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0270323Z Ran 1 test in 1.715s
2022-11-23T02:30:58.0270468Z 
2022-11-23T02:30:58.0270568Z OK
2022-11-23T02:30:58.0270710Z 
2022-11-23T02:30:58.0270844Z Generating XML reports...
2022-11-23T02:30:58.0271444Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20221123022914.xml
2022-11-23T02:30:58.0272131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:58.0272596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:58.0273188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:58.0273670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:58.0273881Z 
2022-11-23T02:30:58.0274004Z Running tests...
2022-11-23T02:30:58.0274423Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0275039Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:30:58.0275501Z   test_set_get (__main__.PrefixFileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:30:58.0275863Z ok (1.686s)
2022-11-23T02:30:58.0276016Z 
2022-11-23T02:30:58.0276286Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0276626Z Ran 1 test in 1.686s
2022-11-23T02:30:58.0276767Z 
2022-11-23T02:30:58.0276865Z OK
2022-11-23T02:30:58.0277005Z 
2022-11-23T02:30:58.0277139Z Generating XML reports...
2022-11-23T02:30:58.0277731Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20221123022918.xml
2022-11-23T02:30:58.0278418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:58.0279056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:58.0279808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:58.0280297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:58.0280510Z 
2022-11-23T02:30:58.0280627Z Running tests...
2022-11-23T02:30:58.0281046Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0281587Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:30:58.0282011Z   test_get_underlying_store (__main__.PrefixStoreTest) ... ok (0.003s)
2022-11-23T02:30:58.0282246Z 
2022-11-23T02:30:58.0282513Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0282849Z Ran 1 test in 0.003s
2022-11-23T02:30:58.0283016Z 
2022-11-23T02:30:58.0283115Z OK
2022-11-23T02:30:58.0283228Z 
2022-11-23T02:30:58.0283366Z Generating XML reports...
2022-11-23T02:30:58.0283946Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixStoreTest-20221123022922.xml
2022-11-23T02:30:58.0284645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:58.0285085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:58.0285677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:58.0286159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:58.0286399Z 
2022-11-23T02:30:58.0286515Z Running tests...
2022-11-23T02:30:58.0286909Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0287599Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:30:58.0288135Z   test_compare_set (__main__.PrefixTCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:30:58.0288661Z ok (1.725s)
2022-11-23T02:30:58.0288817Z 
2022-11-23T02:30:58.0289093Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0289427Z Ran 1 test in 1.725s
2022-11-23T02:30:58.0289594Z 
2022-11-23T02:30:58.0289696Z OK
2022-11-23T02:30:58.0289810Z 
2022-11-23T02:30:58.0289941Z Generating XML reports...
2022-11-23T02:30:58.0290527Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20221123022925.xml
2022-11-23T02:30:58.0291232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:58.0291670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:58.0292259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:58.0292753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:58.0293150Z 
2022-11-23T02:30:58.0293265Z Running tests...
2022-11-23T02:30:58.0293707Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0294223Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:30:58.0294695Z   test_set_get (__main__.PrefixTCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:30:58.0295018Z ok (1.738s)
2022-11-23T02:30:58.0295342Z 
2022-11-23T02:30:58.0295617Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0295956Z Ran 1 test in 1.739s
2022-11-23T02:30:58.0296125Z 
2022-11-23T02:30:58.0296227Z OK
2022-11-23T02:30:58.0296340Z 
2022-11-23T02:30:58.0296471Z Generating XML reports...
2022-11-23T02:30:58.0297284Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20221123022929.xml
2022-11-23T02:30:58.0297996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:58.0298440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:58.0299035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:58.0299518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:58.0299760Z 
2022-11-23T02:30:58.0299875Z Running tests...
2022-11-23T02:30:58.0300428Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0301139Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:30:58.0301621Z   test_set_get (__main__.PythonStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:30:58.0301945Z ok (1.708s)
2022-11-23T02:30:58.0302097Z 
2022-11-23T02:30:58.0302369Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0302705Z Ran 1 test in 1.708s
2022-11-23T02:30:58.0302875Z 
2022-11-23T02:30:58.0302949Z OK
2022-11-23T02:30:58.0303090Z 
2022-11-23T02:30:58.0303224Z Generating XML reports...
2022-11-23T02:30:58.0303806Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20221123022933.xml
2022-11-23T02:30:58.0304503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:58.0304951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:58.0305541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:58.0306024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:58.0306262Z 
2022-11-23T02:30:58.0306379Z Running tests...
2022-11-23T02:30:58.0306769Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0307388Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:30:58.0307881Z   test_nominal (__main__.RendezvousEnvTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:30:58.0308216Z ok (1.682s)
2022-11-23T02:30:58.0308369Z 
2022-11-23T02:30:58.0308797Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0309127Z Ran 1 test in 1.682s
2022-11-23T02:30:58.0309291Z 
2022-11-23T02:30:58.0309544Z OK
2022-11-23T02:30:58.0309685Z 
2022-11-23T02:30:58.0309815Z Generating XML reports...
2022-11-23T02:30:58.0310404Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20221123022937.xml
2022-11-23T02:30:58.0311102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:58.0311543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:58.0312142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:58.0312973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:58.0313298Z 
2022-11-23T02:30:58.0313411Z Running tests...
2022-11-23T02:30:58.0313795Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0314372Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:30:58.0314846Z   test_common_errors (__main__.RendezvousFileTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:30:58.0315175Z ok (1.722s)
2022-11-23T02:30:58.0315331Z 
2022-11-23T02:30:58.0315596Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0315919Z Ran 1 test in 1.722s
2022-11-23T02:30:58.0316084Z 
2022-11-23T02:30:58.0316154Z OK
2022-11-23T02:30:58.0316289Z 
2022-11-23T02:30:58.0316420Z Generating XML reports...
2022-11-23T02:30:58.0316988Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20221123022941.xml
2022-11-23T02:30:58.0317669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:58.0318096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:58.0318673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:58.0319141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:58.0319368Z 
2022-11-23T02:30:58.0319455Z Running tests...
2022-11-23T02:30:58.0319857Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0320374Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:30:58.0320837Z   test_nominal (__main__.RendezvousFileTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:30:58.0321157Z ok (1.685s)
2022-11-23T02:30:58.0321306Z 
2022-11-23T02:30:58.0321574Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0321899Z Ran 1 test in 1.685s
2022-11-23T02:30:58.0322065Z 
2022-11-23T02:30:58.0322135Z OK
2022-11-23T02:30:58.0322268Z 
2022-11-23T02:30:58.0322398Z Generating XML reports...
2022-11-23T02:30:58.0322960Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20221123022945.xml
2022-11-23T02:30:58.0323629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:58.0324053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:58.0324621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:58.0325090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:58.0325320Z 
2022-11-23T02:30:58.0325407Z Running tests...
2022-11-23T02:30:58.0325874Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0326392Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:30:58.0326871Z   test_common_errors (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:30:58.0327195Z ok (1.683s)
2022-11-23T02:30:58.0327344Z 
2022-11-23T02:30:58.0327605Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0327931Z Ran 1 test in 1.683s
2022-11-23T02:30:58.0328094Z 
2022-11-23T02:30:58.0328165Z OK
2022-11-23T02:30:58.0328301Z 
2022-11-23T02:30:58.0328430Z Generating XML reports...
2022-11-23T02:30:58.0328992Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123022949.xml
2022-11-23T02:30:58.0329669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:58.0330091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:58.0330666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:58.0331380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:58.0331616Z 
2022-11-23T02:30:58.0331706Z Running tests...
2022-11-23T02:30:58.0332126Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0332658Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:30:58.0333146Z   test_dns_timeout (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:30:58.0333943Z [W socket.cpp:601] [c10d] The IPv6 network addresses of (dnsnotexist, 23456) cannot be retrieved (gai error: -2 - Name or service not known).
2022-11-23T02:30:58.0334660Z [E socket.cpp:860] [c10d] The client socket has timed out after 1s while trying to connect to (dnsnotexist, 23456).
2022-11-23T02:30:58.0335027Z ok (1.717s)
2022-11-23T02:30:58.0335186Z 
2022-11-23T02:30:58.0335436Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0335778Z Ran 1 test in 1.717s
2022-11-23T02:30:58.0335945Z 
2022-11-23T02:30:58.0336043Z OK
2022-11-23T02:30:58.0336184Z 
2022-11-23T02:30:58.0336316Z Generating XML reports...
2022-11-23T02:30:58.0337105Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123022953.xml
2022-11-23T02:30:58.0337827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:58.0338292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:58.0338861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:58.0339345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:58.0339585Z 
2022-11-23T02:30:58.0339703Z Running tests...
2022-11-23T02:30:58.0340126Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0340795Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:30:58.0341260Z   test_nominal (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:30:58.0341607Z ok (1.746s)
2022-11-23T02:30:58.0341761Z 
2022-11-23T02:30:58.0342002Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0342327Z Ran 1 test in 1.746s
2022-11-23T02:30:58.0342483Z 
2022-11-23T02:30:58.0342584Z OK
2022-11-23T02:30:58.0342718Z 
2022-11-23T02:30:58.0342844Z Generating XML reports...
2022-11-23T02:30:58.0343375Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123022957.xml
2022-11-23T02:30:58.0344255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:58.0344809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:58.0345390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:58.0345876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:58.0346115Z 
2022-11-23T02:30:58.0346234Z Running tests...
2022-11-23T02:30:58.0346991Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0347505Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:30:58.0348010Z   test_tcp_store_timeout_set (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:30:58.0348379Z ok (11.988s)
2022-11-23T02:30:58.0348536Z 
2022-11-23T02:30:58.0348780Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0349119Z Ran 1 test in 11.988s
2022-11-23T02:30:58.0349290Z 
2022-11-23T02:30:58.0349390Z OK
2022-11-23T02:30:58.0349532Z 
2022-11-23T02:30:58.0349664Z Generating XML reports...
2022-11-23T02:30:58.0350556Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123023002.xml
2022-11-23T02:30:58.0351331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:58.0351795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:58.0352358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:58.0352845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:58.0353078Z 
2022-11-23T02:30:58.0353193Z Running tests...
2022-11-23T02:30:58.0353612Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0354122Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:30:58.0354616Z   test_unknown_handler (__main__.RendezvousTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:30:58.0354972Z ok (1.685s)
2022-11-23T02:30:58.0355133Z 
2022-11-23T02:30:58.0355381Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0355721Z Ran 1 test in 1.685s
2022-11-23T02:30:58.0355889Z 
2022-11-23T02:30:58.0355991Z OK
2022-11-23T02:30:58.0356133Z 
2022-11-23T02:30:58.0356264Z Generating XML reports...
2022-11-23T02:30:58.0356807Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20221123023016.xml
2022-11-23T02:30:58.0357825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:58.0358291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:58.0358858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:58.0359344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:58.0359584Z 
2022-11-23T02:30:58.0359703Z Running tests...
2022-11-23T02:30:58.0360120Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0360793Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:30:58.0361269Z   test_url_with_node_params (__main__.RendezvousTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:30:58.0361622Z ok (1.734s)
2022-11-23T02:30:58.0361775Z 
2022-11-23T02:30:58.0362010Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0362336Z Ran 1 test in 1.734s
2022-11-23T02:30:58.0362500Z 
2022-11-23T02:30:58.0362598Z OK
2022-11-23T02:30:58.0362730Z 
2022-11-23T02:30:58.0362832Z Generating XML reports...
2022-11-23T02:30:58.0363384Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20221123023020.xml
2022-11-23T02:30:58.0364106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:58.0364567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:58.0365306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:58.0365791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:58.0366028Z 
2022-11-23T02:30:58.0366143Z Running tests...
2022-11-23T02:30:58.0366561Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0367066Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:30:58.0367728Z   test_address_already_in_use (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:30:58.0368334Z [W socket.cpp:426] [c10d] The server socket has failed to bind to [::]:35805 (errno: 98 - Address already in use).
2022-11-23T02:30:58.0368911Z [W socket.cpp:426] [c10d] The server socket has failed to bind to 0.0.0.0:35805 (errno: 98 - Address already in use).
2022-11-23T02:30:58.0369501Z [E socket.cpp:462] [c10d] The server socket has failed to listen on any local network address.
2022-11-23T02:30:58.0369836Z ok (1.741s)
2022-11-23T02:30:58.0369986Z 
2022-11-23T02:30:58.0370256Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0370562Z Ran 1 test in 1.741s
2022-11-23T02:30:58.0370725Z 
2022-11-23T02:30:58.0370821Z OK
2022-11-23T02:30:58.0370955Z 
2022-11-23T02:30:58.0371082Z Generating XML reports...
2022-11-23T02:30:58.0371605Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123023024.xml
2022-11-23T02:30:58.0372267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:58.0372719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:58.0373295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:58.0373749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:58.0373984Z 
2022-11-23T02:30:58.0374097Z Running tests...
2022-11-23T02:30:58.0374682Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0375191Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:30:58.0375667Z   test_compare_set (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:30:58.0376020Z ok (1.761s)
2022-11-23T02:30:58.0376178Z 
2022-11-23T02:30:58.0376445Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0376947Z Ran 1 test in 1.761s
2022-11-23T02:30:58.0377138Z 
2022-11-23T02:30:58.0377253Z OK
2022-11-23T02:30:58.0377396Z 
2022-11-23T02:30:58.0377532Z Generating XML reports...
2022-11-23T02:30:58.0378083Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123023028.xml
2022-11-23T02:30:58.0378775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:58.0379243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:58.0379828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:58.0380290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:58.0380528Z 
2022-11-23T02:30:58.0380646Z Running tests...
2022-11-23T02:30:58.0381068Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0381581Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:30:58.0382090Z   test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:30:58.0382855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:58.0383718Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:30:58.0384105Z ok (1.765s)
2022-11-23T02:30:58.0384264Z 
2022-11-23T02:30:58.0384534Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0384871Z Ran 1 test in 1.765s
2022-11-23T02:30:58.0385039Z 
2022-11-23T02:30:58.0385113Z OK
2022-11-23T02:30:58.0385251Z 
2022-11-23T02:30:58.0385382Z Generating XML reports...
2022-11-23T02:30:58.0385944Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123023032.xml
2022-11-23T02:30:58.0386636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:58.0387080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:58.0387841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:58.0388379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:58.0388784Z 
2022-11-23T02:30:58.0388875Z Running tests...
2022-11-23T02:30:58.0389295Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0389826Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:30:58.0390336Z   test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:30:58.0390687Z ok (1.731s)
2022-11-23T02:30:58.0390842Z 
2022-11-23T02:30:58.0391116Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0391455Z Ran 1 test in 1.731s
2022-11-23T02:30:58.0391625Z 
2022-11-23T02:30:58.0391699Z OK
2022-11-23T02:30:58.0391841Z 
2022-11-23T02:30:58.0391974Z Generating XML reports...
2022-11-23T02:30:58.0392536Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123023037.xml
2022-11-23T02:30:58.0393384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:58.0393992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:58.0394579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:58.0395059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:58.0395294Z 
2022-11-23T02:30:58.0395384Z Running tests...
2022-11-23T02:30:58.0395804Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0396339Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:30:58.0396854Z   test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:30:58.0397206Z ok (1.737s)
2022-11-23T02:30:58.0397365Z 
2022-11-23T02:30:58.0397635Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0397968Z Ran 1 test in 1.737s
2022-11-23T02:30:58.0398137Z 
2022-11-23T02:30:58.0398210Z OK
2022-11-23T02:30:58.0398514Z 
2022-11-23T02:30:58.0398642Z Generating XML reports...
2022-11-23T02:30:58.0399193Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123023041.xml
2022-11-23T02:30:58.0399859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:58.0400283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:58.0400851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:58.0401567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:58.0401809Z 
2022-11-23T02:30:58.0401899Z Running tests...
2022-11-23T02:30:58.0402320Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0402853Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:30:58.0403337Z   test_multitenancy (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:30:58.0403669Z ok (1.745s)
2022-11-23T02:30:58.0403988Z 
2022-11-23T02:30:58.0404252Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0404577Z Ran 1 test in 1.745s
2022-11-23T02:30:58.0404737Z 
2022-11-23T02:30:58.0404807Z OK
2022-11-23T02:30:58.0404941Z 
2022-11-23T02:30:58.0405070Z Generating XML reports...
2022-11-23T02:30:58.0405613Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123023045.xml
2022-11-23T02:30:58.0406432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:58.0406900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:58.0407554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:58.0408038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:58.0408277Z 
2022-11-23T02:30:58.0408366Z Running tests...
2022-11-23T02:30:58.0408779Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0409458Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:30:58.0410090Z   test_numkeys_delkeys (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:30:58.0410443Z ok (3.716s)
2022-11-23T02:30:58.0410601Z 
2022-11-23T02:30:58.0410872Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0411211Z Ran 1 test in 3.716s
2022-11-23T02:30:58.0411356Z 
2022-11-23T02:30:58.0411456Z OK
2022-11-23T02:30:58.0411595Z 
2022-11-23T02:30:58.0411729Z Generating XML reports...
2022-11-23T02:30:58.0412298Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123023049.xml
2022-11-23T02:30:58.0413116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:58.0413569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:58.0414134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:58.0414666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:58.0414895Z 
2022-11-23T02:30:58.0414981Z Running tests...
2022-11-23T02:30:58.0415390Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0415911Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:30:58.0416341Z   test_set_get (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:30:58.0416669Z ok (1.688s)
2022-11-23T02:30:58.0417202Z 
2022-11-23T02:30:58.0417480Z ----------------------------------------------------------------------
2022-11-23T02:30:58.0417810Z Ran 1 test in 1.689s
2022-11-23T02:30:58.0417953Z 
2022-11-23T02:30:58.0418049Z OK
2022-11-23T02:30:58.0418192Z 
2022-11-23T02:30:58.0418323Z Generating XML reports...
2022-11-23T02:30:58.0418880Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123023055.xml
2022-11-23T02:30:58.0419205Z 
2022-11-23T02:30:58.0419603Z ##[endgroup]
2022-11-23T02:30:58.0420167Z FINISHED PRINTING LOG FILE of distributed/test_store (/var/lib/jenkins/workspace/test/test-reports/distributed-test_store_wcuh6d55)
2022-11-23T02:30:58.0420492Z 
2022-11-23T02:30:58.4011286Z 
2022-11-23T02:30:58.4011816Z real	2m17.352s
2022-11-23T02:30:58.4012128Z user	2m52.140s
2022-11-23T02:30:58.4012772Z sys	2m34.025s
2022-11-23T02:30:58.4013387Z + python test/run_test.py --verbose -i distributed/test_pg_wrapper
2022-11-23T02:31:00.7603507Z Ignoring disabled issues:  []
2022-11-23T02:31:00.8128368Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T02:31:00.8128966Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T02:31:00.8129307Z Selected tests:
2022-11-23T02:31:00.8129579Z  distributed/test_pg_wrapper
2022-11-23T02:31:00.8156888Z Prioritized test from test file changes.
2022-11-23T02:31:00.8157214Z reordering tests for PR:
2022-11-23T02:31:00.8157499Z prioritized: []
2022-11-23T02:31:00.8157986Z the rest: ['distributed/test_pg_wrapper']
2022-11-23T02:31:00.8158196Z 
2022-11-23T02:31:00.8158765Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T02:31:00.8159702Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T02:31:00.8166426Z parallel (file granularity) tests:
2022-11-23T02:31:00.8166726Z  
2022-11-23T02:31:00.8166987Z serial (file granularity) tests:
2022-11-23T02:31:00.8167258Z  distributed/test_pg_wrapper
2022-11-23T02:31:03.1407426Z Ignoring disabled issues:  []
2022-11-23T02:31:03.1409361Z Ignoring disabled issues:  []
2022-11-23T02:31:03.6050059Z Running distributed/test_pg_wrapper ... [2022-11-23 02:31:03.604428]
2022-11-23T02:31:03.6051710Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_pg_wrapper.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:31:03.604887]
2022-11-23T02:32:57.1387178Z 
2022-11-23T02:32:57.1388109Z Expand the folded group to see the log file of distributed/test_pg_wrapper
2022-11-23T02:32:57.1390710Z ##[group]PRINTING LOG FILE of distributed/test_pg_wrapper (/var/lib/jenkins/workspace/test/test-reports/distributed-test_pg_wrapper_uzl5xeky)
2022-11-23T02:32:57.1391224Z <unittest.suite.TestSuite tests=[]>
2022-11-23T02:32:57.1391557Z <unittest.suite.TestSuite tests=[]>
2022-11-23T02:32:57.1393063Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_hang>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_cuda>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_cuda_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_cuda>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_cuda_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_debug_mode>]>
2022-11-23T02:32:57.1394524Z test_collective_hang (__main__.ProcessGroupGlooWrapperTest)
2022-11-23T02:32:57.1394966Z test_collective_shape_mismatch (__main__.ProcessGroupGlooWrapperTest)
2022-11-23T02:32:57.1395479Z test_collective_shape_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest)
2022-11-23T02:32:57.1396236Z test_collective_shape_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2022-11-23T02:32:57.1397257Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2022-11-23T02:32:57.1398044Z test_collectives_op_mismatch (__main__.ProcessGroupGlooWrapperTest)
2022-11-23T02:32:57.1399037Z test_collectives_op_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest)
2022-11-23T02:32:57.1399591Z test_collectives_op_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2022-11-23T02:32:57.1400277Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2022-11-23T02:32:57.1401327Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_hang>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_shape_mismatch>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_shape_mismatch_debug_mode>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collectives_op_mismatch>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collectives_op_mismatch_debug_mode>]>
2022-11-23T02:32:57.1402790Z test_collective_hang (__main__.ProcessGroupNCCLWrapperTest)
2022-11-23T02:32:57.1403729Z test_collective_shape_mismatch (__main__.ProcessGroupNCCLWrapperTest)
2022-11-23T02:32:57.1404312Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest)
2022-11-23T02:32:57.1404780Z test_collectives_op_mismatch (__main__.ProcessGroupNCCLWrapperTest)
2022-11-23T02:32:57.1405219Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest)
2022-11-23T02:32:57.1406243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1407042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1407656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1408113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1408343Z 
2022-11-23T02:32:57.1408457Z Running tests...
2022-11-23T02:32:57.1408877Z ----------------------------------------------------------------------
2022-11-23T02:32:57.1409385Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-11-23T02:32:57.1409914Z   test_collective_hang (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:32:57.1410422Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67288
2022-11-23T02:32:57.1410877Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67289
2022-11-23T02:32:57.1411309Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 67290
2022-11-23T02:32:57.1411753Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 67291
2022-11-23T02:32:57.1412368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1413031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1413603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1414091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1414662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1415098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1415682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1416152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1417057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1417517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1418114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1418579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1419163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1419591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1420281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1420779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1421204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:57.1421692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:32:57.1422157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:57.1422612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:32:57.1423095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:57.1423583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:57.1424073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:32:57.1424654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:32:57.1425300Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1425988Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1426672Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1427329Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1427845Z [E ProcessGroupGloo.cpp:2802] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-11-23T02:32:57.1428325Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 2000 ms
2022-11-23T02:32:57.1428913Z [E ProcessGroupGloo.cpp:137] Rank 2 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2022-11-23T02:32:57.1429577Z [E ProcessGroupGloo.cpp:137] Rank 3 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2022-11-23T02:32:57.1430026Z ok (4.339s)
2022-11-23T02:32:57.1430175Z 
2022-11-23T02:32:57.1430454Z ----------------------------------------------------------------------
2022-11-23T02:32:57.1430781Z Ran 1 test in 4.339s
2022-11-23T02:32:57.1430923Z 
2022-11-23T02:32:57.1431017Z OK
2022-11-23T02:32:57.1431153Z 
2022-11-23T02:32:57.1431278Z Generating XML reports...
2022-11-23T02:32:57.1431906Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123023107.xml
2022-11-23T02:32:57.1432626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1433072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1433647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1434112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1434339Z 
2022-11-23T02:32:57.1434429Z Running tests...
2022-11-23T02:32:57.1434828Z ----------------------------------------------------------------------
2022-11-23T02:32:57.1435356Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-11-23T02:32:57.1435888Z   test_collective_shape_mismatch (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:32:57.1436379Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67655
2022-11-23T02:32:57.1436892Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67656
2022-11-23T02:32:57.1437350Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 67657
2022-11-23T02:32:57.1437783Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 67658
2022-11-23T02:32:57.1438388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1438838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1439415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1439864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1440444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1440888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1441445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1441983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1442559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1442999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1443553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1444016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1444586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1445008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1445587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1446048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1446484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:32:57.1446934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:32:57.1447387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:57.1447849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:57.1448332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:57.1448807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:57.1449298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:32:57.1449782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:32:57.1450426Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1451113Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1451796Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1452473Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1452847Z ok (4.347s)
2022-11-23T02:32:57.1452992Z 
2022-11-23T02:32:57.1453259Z ----------------------------------------------------------------------
2022-11-23T02:32:57.1453583Z Ran 1 test in 4.347s
2022-11-23T02:32:57.1453803Z 
2022-11-23T02:32:57.1453899Z OK
2022-11-23T02:32:57.1454016Z 
2022-11-23T02:32:57.1454140Z Generating XML reports...
2022-11-23T02:32:57.1454770Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123023114.xml
2022-11-23T02:32:57.1455499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1455929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1456502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1457287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1457520Z 
2022-11-23T02:32:57.1457631Z Running tests...
2022-11-23T02:32:57.1458028Z ----------------------------------------------------------------------
2022-11-23T02:32:57.1458565Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-11-23T02:32:57.1459105Z   test_collective_shape_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:32:57.1459714Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68022
2022-11-23T02:32:57.1460163Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68023
2022-11-23T02:32:57.1460602Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68024
2022-11-23T02:32:57.1461044Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68025
2022-11-23T02:32:57.1461642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1462092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1462659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1463113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1463693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1464190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1464764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1465211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1465787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1466223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1466792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1467239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1467813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1468258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1468805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1469264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1469902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:57.1470374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:32:57.1470818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:32:57.1471272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:57.1471840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:57.1472330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:32:57.1472814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:57.1473453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:32:57.1474089Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1474734Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1475395Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1476052Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1476739Z ok (6.144s)
2022-11-23T02:32:57.1476870Z 
2022-11-23T02:32:57.1477145Z ----------------------------------------------------------------------
2022-11-23T02:32:57.1477469Z Ran 1 test in 6.144s
2022-11-23T02:32:57.1477632Z 
2022-11-23T02:32:57.1477724Z OK
2022-11-23T02:32:57.1477856Z 
2022-11-23T02:32:57.1477963Z Generating XML reports...
2022-11-23T02:32:57.1478587Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123023121.xml
2022-11-23T02:32:57.1479310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1479758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1480315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1480783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1481169Z 
2022-11-23T02:32:57.1481272Z Running tests...
2022-11-23T02:32:57.1481642Z ----------------------------------------------------------------------
2022-11-23T02:32:57.1482340Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-11-23T02:32:57.1482893Z   test_collective_shape_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:32:57.1483415Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68393
2022-11-23T02:32:57.1483843Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68394
2022-11-23T02:32:57.1484278Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68395
2022-11-23T02:32:57.1484868Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68396
2022-11-23T02:32:57.1485462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1485881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1486431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1486877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1487413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1487836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1488381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1489010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1489642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1490099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1490677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1491117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1491692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1492286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1492829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1493253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1493671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:57.1494132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:32:57.1494653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:32:57.1495083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:57.1495546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:57.1496017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:57.1496471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:32:57.1497396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:32:57.1498069Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1498761Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1499491Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1500173Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1500700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:32:57.1501190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T02:32:57.1501657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:32:57.1502142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-11-23T02:32:57.1502787Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T02:32:57.1503475Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T02:32:57.1504136Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T02:32:57.1504807Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T02:32:57.1505192Z ok (6.365s)
2022-11-23T02:32:57.1505337Z 
2022-11-23T02:32:57.1505590Z ----------------------------------------------------------------------
2022-11-23T02:32:57.1505968Z Ran 1 test in 6.365s
2022-11-23T02:32:57.1506291Z 
2022-11-23T02:32:57.1506382Z OK
2022-11-23T02:32:57.1506509Z 
2022-11-23T02:32:57.1506630Z Generating XML reports...
2022-11-23T02:32:57.1507317Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123023129.xml
2022-11-23T02:32:57.1508041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1508485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1509193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1509668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1509898Z 
2022-11-23T02:32:57.1510003Z Running tests...
2022-11-23T02:32:57.1510403Z ----------------------------------------------------------------------
2022-11-23T02:32:57.1510914Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-11-23T02:32:57.1511461Z   test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:32:57.1512138Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68776
2022-11-23T02:32:57.1512554Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68777
2022-11-23T02:32:57.1513080Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68778
2022-11-23T02:32:57.1513498Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68779
2022-11-23T02:32:57.1514088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1514506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1515058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1515515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1516074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1516485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1517040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1517488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1518026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1518451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1519184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1519640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1520199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1520646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1521214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1521659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1522100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:57.1522574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:57.1523200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:32:57.1523632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:32:57.1524096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:57.1524637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:57.1525127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:32:57.1525583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:32:57.1526227Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1526901Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1527735Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1528415Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1528954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:32:57.1529453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T02:32:57.1530313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-11-23T02:32:57.1530809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:32:57.1531475Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T02:32:57.1532166Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T02:32:57.1532824Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T02:32:57.1533658Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T02:32:57.1534218Z ok (4.466s)
2022-11-23T02:32:57.1534371Z 
2022-11-23T02:32:57.1534645Z ----------------------------------------------------------------------
2022-11-23T02:32:57.1534959Z Ran 1 test in 4.466s
2022-11-23T02:32:57.1535122Z 
2022-11-23T02:32:57.1535217Z OK
2022-11-23T02:32:57.1535351Z 
2022-11-23T02:32:57.1535477Z Generating XML reports...
2022-11-23T02:32:57.1536093Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123023138.xml
2022-11-23T02:32:57.1537273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1537752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1538347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1538801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1539033Z 
2022-11-23T02:32:57.1539148Z Running tests...
2022-11-23T02:32:57.1539556Z ----------------------------------------------------------------------
2022-11-23T02:32:57.1540076Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-11-23T02:32:57.1540773Z   test_collectives_op_mismatch (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:32:57.1541267Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69155
2022-11-23T02:32:57.1541706Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69156
2022-11-23T02:32:57.1542125Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 69157
2022-11-23T02:32:57.1542560Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 69158
2022-11-23T02:32:57.1543156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1543663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1544239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1544692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1545439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1545870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1546450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1546905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1547485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1547908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1548492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1549194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1549733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1550171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1550729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1551185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1551589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:57.1552206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:32:57.1552678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:32:57.1553126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:57.1553613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:57.1554101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:57.1554590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:32:57.1555054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:32:57.1555702Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1556389Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1557208Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1557849Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1558227Z ok (4.467s)
2022-11-23T02:32:57.1558367Z 
2022-11-23T02:32:57.1558625Z ----------------------------------------------------------------------
2022-11-23T02:32:57.1558927Z Ran 1 test in 4.468s
2022-11-23T02:32:57.1559082Z 
2022-11-23T02:32:57.1559169Z OK
2022-11-23T02:32:57.1559297Z 
2022-11-23T02:32:57.1559415Z Generating XML reports...
2022-11-23T02:32:57.1560019Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123023145.xml
2022-11-23T02:32:57.1560888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1561402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1561978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1562428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1562655Z 
2022-11-23T02:32:57.1562760Z Running tests...
2022-11-23T02:32:57.1563351Z ----------------------------------------------------------------------
2022-11-23T02:32:57.1564079Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-11-23T02:32:57.1564582Z   test_collectives_op_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:32:57.1565249Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69522
2022-11-23T02:32:57.1565695Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69523
2022-11-23T02:32:57.1566143Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 69524
2022-11-23T02:32:57.1566567Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 69525
2022-11-23T02:32:57.1567239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1567687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1568388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1568837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1569390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1570000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1570554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1571038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1571622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1572048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1572626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1573100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1573829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1574234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1574782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1575228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1575650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:32:57.1576092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:32:57.1576537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:57.1577486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:57.1577964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:57.1578469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:32:57.1578964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:57.1579542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:32:57.1580215Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1581070Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1581736Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1582395Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1582751Z ok (6.153s)
2022-11-23T02:32:57.1582901Z 
2022-11-23T02:32:57.1583163Z ----------------------------------------------------------------------
2022-11-23T02:32:57.1583482Z Ran 1 test in 6.154s
2022-11-23T02:32:57.1583640Z 
2022-11-23T02:32:57.1583711Z OK
2022-11-23T02:32:57.1583841Z 
2022-11-23T02:32:57.1583965Z Generating XML reports...
2022-11-23T02:32:57.1584582Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123023152.xml
2022-11-23T02:32:57.1585375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1585789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1586343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1586795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1587019Z 
2022-11-23T02:32:57.1587104Z Running tests...
2022-11-23T02:32:57.1587493Z ----------------------------------------------------------------------
2022-11-23T02:32:57.1588003Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-11-23T02:32:57.1588538Z   test_collectives_op_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:32:57.1589205Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69893
2022-11-23T02:32:57.1589656Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69894
2022-11-23T02:32:57.1590096Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 69895
2022-11-23T02:32:57.1590518Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 69896
2022-11-23T02:32:57.1591125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1591590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1592162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1592610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1593184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1593627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1594195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1594644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1595216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1595800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1596330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1596773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1597426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1597863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1598398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1598847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1599265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:32:57.1599700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:57.1600153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:32:57.1600596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:57.1601247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:57.1601721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:57.1602268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:32:57.1602926Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1603597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:32:57.1604275Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1605037Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1605855Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1606697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:32:57.1607255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:32:57.1607793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T02:32:57.1608315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-11-23T02:32:57.1609072Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T02:32:57.1610005Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T02:32:57.1628367Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T02:32:57.1629203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T02:32:57.1629604Z ok (6.347s)
2022-11-23T02:32:57.1629763Z 
2022-11-23T02:32:57.1630018Z ----------------------------------------------------------------------
2022-11-23T02:32:57.1630512Z Ran 1 test in 6.347s
2022-11-23T02:32:57.1630670Z 
2022-11-23T02:32:57.1630936Z OK
2022-11-23T02:32:57.1631071Z 
2022-11-23T02:32:57.1631197Z Generating XML reports...
2022-11-23T02:32:57.1631816Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123023200.xml
2022-11-23T02:32:57.1632554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1633009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1633568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1634203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1634455Z 
2022-11-23T02:32:57.1634570Z Running tests...
2022-11-23T02:32:57.1634986Z ----------------------------------------------------------------------
2022-11-23T02:32:57.1635506Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-11-23T02:32:57.1636069Z   test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:32:57.1636600Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70276
2022-11-23T02:32:57.1637037Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70277
2022-11-23T02:32:57.1637493Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 70278
2022-11-23T02:32:57.1638094Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 70279
2022-11-23T02:32:57.1638707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1639126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1639784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1640432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1641014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1641439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1642010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1642475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1643039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1643484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1644054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1644516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1645076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1645524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1646094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1646538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1646975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:57.1647455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:57.1647927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:32:57.1648716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:32:57.1649200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:57.1649699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:32:57.1650365Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1650880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:57.1651373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:32:57.1652092Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1653095Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1653784Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:32:57.1654314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:32:57.1654807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:32:57.1655282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-11-23T02:32:57.1655778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T02:32:57.1656428Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T02:32:57.1657465Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T02:32:57.1658243Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T02:32:57.1658927Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T02:32:57.1659319Z ok (4.576s)
2022-11-23T02:32:57.1659470Z 
2022-11-23T02:32:57.1659740Z ----------------------------------------------------------------------
2022-11-23T02:32:57.1660051Z Ran 1 test in 4.576s
2022-11-23T02:32:57.1660212Z 
2022-11-23T02:32:57.1660457Z OK
2022-11-23T02:32:57.1660588Z 
2022-11-23T02:32:57.1660710Z Generating XML reports...
2022-11-23T02:32:57.1661493Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123023209.xml
2022-11-23T02:32:57.1662231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1662689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1663272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1663728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1664001Z 
2022-11-23T02:32:57.1664112Z Running tests...
2022-11-23T02:32:57.1664521Z ----------------------------------------------------------------------
2022-11-23T02:32:57.1665032Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-11-23T02:32:57.1665555Z   test_collective_hang (__main__.ProcessGroupNCCLWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:32:57.1666054Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70655
2022-11-23T02:32:57.1666515Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70656
2022-11-23T02:32:57.1667113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1667562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1668136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1668744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1669306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1669736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1670471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1671006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1671463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:57.1671960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:57.1672446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:57.1672905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:57.1673568Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:57.1674406Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:57.1674893Z [E ProcessGroupGloo.cpp:2802] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-11-23T02:32:57.1675531Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 2000 ms
2022-11-23T02:32:57.1675937Z ok (4.138s)
2022-11-23T02:32:57.1676089Z 
2022-11-23T02:32:57.1676359Z ----------------------------------------------------------------------
2022-11-23T02:32:57.1676666Z Ran 1 test in 4.139s
2022-11-23T02:32:57.1676829Z 
2022-11-23T02:32:57.1676923Z OK
2022-11-23T02:32:57.1677058Z 
2022-11-23T02:32:57.1677183Z Generating XML reports...
2022-11-23T02:32:57.1677796Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221123023216.xml
2022-11-23T02:32:57.1678844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1679296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1679872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1680334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1680566Z 
2022-11-23T02:32:57.1680679Z Running tests...
2022-11-23T02:32:57.1681089Z ----------------------------------------------------------------------
2022-11-23T02:32:57.1681600Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-11-23T02:32:57.1682137Z   test_collective_shape_mismatch (__main__.ProcessGroupNCCLWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:32:57.1682649Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70864
2022-11-23T02:32:57.1683103Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70865
2022-11-23T02:32:57.1683693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1684148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1684878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1685338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1685876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1686305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1686860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1687287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1687717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:57.1688177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:57.1688709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:57.1689182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:57.1690019Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:57.1690711Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:57.1691085Z ok (5.966s)
2022-11-23T02:32:57.1691234Z 
2022-11-23T02:32:57.1691500Z ----------------------------------------------------------------------
2022-11-23T02:32:57.1691826Z Ran 1 test in 5.966s
2022-11-23T02:32:57.1691989Z 
2022-11-23T02:32:57.1692083Z OK
2022-11-23T02:32:57.1692197Z 
2022-11-23T02:32:57.1692323Z Generating XML reports...
2022-11-23T02:32:57.1693101Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221123023223.xml
2022-11-23T02:32:57.1693815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1694307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1694866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1695322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1695547Z 
2022-11-23T02:32:57.1695653Z Running tests...
2022-11-23T02:32:57.1696022Z ----------------------------------------------------------------------
2022-11-23T02:32:57.1696546Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-11-23T02:32:57.1697594Z   test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:32:57.1698128Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71089
2022-11-23T02:32:57.1698573Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71090
2022-11-23T02:32:57.1699199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1699650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1700206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1700825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1701388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1701819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1702526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1702997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1703443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:57.1703905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:57.1704395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:57.1705056Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:57.1705592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:57.1706285Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:57.1706976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:32:57.1707550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:32:57.1708207Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:32:57.1708844Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:32:57.1709227Z ok (5.927s)
2022-11-23T02:32:57.1709381Z 
2022-11-23T02:32:57.1709817Z ----------------------------------------------------------------------
2022-11-23T02:32:57.1710134Z Ran 1 test in 5.927s
2022-11-23T02:32:57.1710302Z 
2022-11-23T02:32:57.1710401Z OK
2022-11-23T02:32:57.1710541Z 
2022-11-23T02:32:57.1710672Z Generating XML reports...
2022-11-23T02:32:57.1711319Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221123023231.xml
2022-11-23T02:32:57.1712043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1712664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1713320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1713763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1713993Z 
2022-11-23T02:32:57.1714104Z Running tests...
2022-11-23T02:32:57.1714507Z ----------------------------------------------------------------------
2022-11-23T02:32:57.1715033Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-11-23T02:32:57.1715536Z   test_collectives_op_mismatch (__main__.ProcessGroupNCCLWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:32:57.1716045Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71324
2022-11-23T02:32:57.1716500Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71325
2022-11-23T02:32:57.1717106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1717535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1718106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1718762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1719325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1719787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1720376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1720853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1721282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:57.1721772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:57.1722276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:57.1722765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:57.1723586Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:57.1724265Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:57.1724658Z ok (7.048s)
2022-11-23T02:32:57.1724808Z 
2022-11-23T02:32:57.1725048Z ----------------------------------------------------------------------
2022-11-23T02:32:57.1725384Z Ran 1 test in 7.048s
2022-11-23T02:32:57.1725546Z 
2022-11-23T02:32:57.1725719Z OK
2022-11-23T02:32:57.1725863Z 
2022-11-23T02:32:57.1725965Z Generating XML reports...
2022-11-23T02:32:57.1726598Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221123023239.xml
2022-11-23T02:32:57.1727318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1727937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1728502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1728987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1729226Z 
2022-11-23T02:32:57.1729342Z Running tests...
2022-11-23T02:32:57.1729761Z ----------------------------------------------------------------------
2022-11-23T02:32:57.1730286Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-11-23T02:32:57.1731175Z   test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:32:57.1731781Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71550
2022-11-23T02:32:57.1732223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71551
2022-11-23T02:32:57.1732842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1733305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1733892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1734501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1735259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:32:57.1735720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:32:57.1736285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:32:57.1736971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:32:57.1737438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:32:57.1737928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:32:57.1738409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:32:57.1738922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:32:57.1739605Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:57.1740468Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:32:57.1740977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:32:57.1741471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:32:57.1742115Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:32:57.1742765Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:32:57.1743168Z ok (7.137s)
2022-11-23T02:32:57.1743318Z 
2022-11-23T02:32:57.1743582Z ----------------------------------------------------------------------
2022-11-23T02:32:57.1743915Z Ran 1 test in 7.137s
2022-11-23T02:32:57.1744055Z 
2022-11-23T02:32:57.1744151Z OK
2022-11-23T02:32:57.1744285Z 
2022-11-23T02:32:57.1744499Z Generating XML reports...
2022-11-23T02:32:57.1745133Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221123023249.xml
2022-11-23T02:32:57.1745508Z 
2022-11-23T02:32:57.1746016Z ##[endgroup]
2022-11-23T02:32:57.1746597Z FINISHED PRINTING LOG FILE of distributed/test_pg_wrapper (/var/lib/jenkins/workspace/test/test-reports/distributed-test_pg_wrapper_uzl5xeky)
2022-11-23T02:32:57.1746943Z 
2022-11-23T02:32:57.4947522Z 
2022-11-23T02:32:57.4947978Z real	1m59.094s
2022-11-23T02:32:57.4948300Z user	4m13.555s
2022-11-23T02:32:57.4948528Z sys	3m16.146s
2022-11-23T02:32:57.4949094Z + python test/run_test.py --verbose -i distributed/rpc/cuda/test_tensorpipe_agent
2022-11-23T02:32:59.8895829Z Ignoring disabled issues:  []
2022-11-23T02:32:59.9414720Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T02:32:59.9415343Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T02:32:59.9415696Z Selected tests:
2022-11-23T02:32:59.9416358Z  distributed/rpc/cuda/test_tensorpipe_agent
2022-11-23T02:32:59.9441900Z Prioritized test from test file changes.
2022-11-23T02:32:59.9442226Z reordering tests for PR:
2022-11-23T02:32:59.9442532Z prioritized: []
2022-11-23T02:32:59.9443075Z the rest: ['distributed/rpc/cuda/test_tensorpipe_agent']
2022-11-23T02:32:59.9443309Z 
2022-11-23T02:32:59.9443859Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T02:32:59.9444786Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T02:32:59.9450896Z parallel (file granularity) tests:
2022-11-23T02:32:59.9451203Z  
2022-11-23T02:32:59.9451473Z serial (file granularity) tests:
2022-11-23T02:32:59.9451796Z  distributed/rpc/cuda/test_tensorpipe_agent
2022-11-23T02:33:02.2428728Z Ignoring disabled issues:  []
2022-11-23T02:33:02.2893317Z Ignoring disabled issues:  []
2022-11-23T02:33:02.7722137Z Running distributed/rpc/cuda/test_tensorpipe_agent ... [2022-11-23 02:33:02.771518]
2022-11-23T02:33:02.7723046Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/cuda/test_tensorpipe_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:33:02.771952]
2022-11-23T02:55:36.9457999Z 
2022-11-23T02:55:36.9458594Z Expand the folded group to see the log file of distributed/rpc/cuda/test_tensorpipe_agent
2022-11-23T02:55:36.9461254Z ##[group]PRINTING LOG FILE of distributed/rpc/cuda/test_tensorpipe_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-cuda-test_tensorpipe_agent_oynlluog)
2022-11-23T02:55:36.9464626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptxftjtq7
2022-11-23T02:55:36.9465217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptxftjtq7/_remote_module_non_scriptable.py
2022-11-23T02:55:36.9467384Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaDdpComparisonTest testMethod=test_ddp_dist_autograd_local_vs_remote_gpu>]>
2022-11-23T02:55:36.9468028Z test_ddp_dist_autograd_local_vs_remote_gpu (__main__.TensorPipeCudaDdpComparisonTest)
2022-11-23T02:55:36.9468840Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_simple>, <__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_to_cpu_continuation>, <__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_to_cpu_continuation_gpu_root>]>
2022-11-23T02:55:36.9469574Z test_gpu_simple (__main__.TensorPipeCudaDistAutogradTest)
2022-11-23T02:55:36.9470029Z test_gpu_to_cpu_continuation (__main__.TensorPipeCudaDistAutogradTest)
2022-11-23T02:55:36.9470499Z test_gpu_to_cpu_continuation_gpu_root (__main__.TensorPipeCudaDistAutogradTest)
2022-11-23T02:55:36.9471822Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaRemoteModuleTest testMethod=test_input_moved_to_cuda_device>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_input_moved_to_cuda_device_script>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_invalid_devices>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_valid_device>]>
2022-11-23T02:55:36.9472729Z test_input_moved_to_cuda_device (__main__.TensorPipeCudaRemoteModuleTest)
2022-11-23T02:55:36.9473216Z test_input_moved_to_cuda_device_script (__main__.TensorPipeCudaRemoteModuleTest)
2022-11-23T02:55:36.9473671Z test_invalid_devices (__main__.TensorPipeCudaRemoteModuleTest)
2022-11-23T02:55:36.9474090Z test_valid_device (__main__.TensorPipeCudaRemoteModuleTest)
2022-11-23T02:55:36.9474552Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaRpcTest testMethod=test_profiler_remote_cuda>]>
2022-11-23T02:55:36.9475035Z test_profiler_remote_cuda (__main__.TensorPipeCudaRpcTest)
2022-11-23T02:55:36.9476335Z <unittest.suite.TestSuite tests=[<__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_always>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_except_last>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_never>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_never_find_unused>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_always>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_except_last>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_never>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_never_find_unused>]>
2022-11-23T02:55:36.9477722Z test_basic_gloo_ckpt_always (__main__.TensorPipePipeWithDDPTest)
2022-11-23T02:55:36.9478263Z test_basic_gloo_ckpt_except_last (__main__.TensorPipePipeWithDDPTest)
2022-11-23T02:55:36.9478666Z test_basic_gloo_ckpt_never (__main__.TensorPipePipeWithDDPTest)
2022-11-23T02:55:36.9479128Z test_basic_gloo_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest)
2022-11-23T02:55:36.9479578Z test_basic_nccl_ckpt_always (__main__.TensorPipePipeWithDDPTest)
2022-11-23T02:55:36.9479998Z test_basic_nccl_ckpt_except_last (__main__.TensorPipePipeWithDDPTest)
2022-11-23T02:55:36.9480428Z test_basic_nccl_ckpt_never (__main__.TensorPipePipeWithDDPTest)
2022-11-23T02:55:36.9480926Z test_basic_nccl_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest)
2022-11-23T02:55:36.9495264Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_async_execution_nested_with_cuda_future>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_async_execution_with_cuda_future>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_callback_changes_devices>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_custom_class_with_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_list_with_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_list_with_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_int>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_str>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_not_cuda>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_modify_tensor_inplace>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_replace_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_value_on_bad_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_multi>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_nested>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_nested_multi>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu_to_gpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu_to_gpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_default_to_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_6>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_7>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_8>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_6>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_7>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_8>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_non_default_to_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_to_cpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_to_cpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_in_options>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_max_local_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_max_remote_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_min_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_many_to_one>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_loop>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_not_timeout>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_remote>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_remote_response>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_response>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_response_loop>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_multi_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_multi_gpu_self>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_one_to_many>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_remote>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_return_to_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_return_to_gpu_self>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_wrong_worker_name>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_mismatch>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_devices_option_mismatch>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_devices_option_mismatch_reverse>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_with_unpickleable_attributes>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_tensor_view_as_return_value>]>
2022-11-23T02:55:36.9509665Z test_async_execution_nested_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9510113Z test_async_execution_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9510638Z test_cuda_future_callback_changes_devices (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9511210Z test_cuda_future_can_extract_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9511730Z test_cuda_future_can_extract_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9512311Z test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9512885Z test_cuda_future_can_extract_custom_class_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9513477Z test_cuda_future_can_extract_list_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9514040Z test_cuda_future_can_extract_list_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9514665Z test_cuda_future_device_as_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9515142Z test_cuda_future_device_as_int (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9515637Z test_cuda_future_device_as_str (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9516146Z test_cuda_future_device_not_cuda (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9516674Z test_cuda_future_modify_tensor_inplace (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9517177Z test_cuda_future_replace_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9517676Z test_cuda_future_value_on_bad_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9518165Z test_custom_stream (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9518650Z test_custom_stream_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9519106Z test_custom_stream_nested (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9519612Z test_custom_stream_nested_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9520099Z test_device_map_cpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9520647Z test_device_map_cpu_to_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9521113Z test_device_map_cpu_to_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9521706Z test_device_map_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9522222Z test_device_map_gpu_default_to_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9522740Z test_device_map_gpu_mixed_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9523212Z test_device_map_gpu_mixed_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9523697Z test_device_map_gpu_mixed_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9524167Z test_device_map_gpu_mixed_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9524636Z test_device_map_gpu_mixed_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9525130Z test_device_map_gpu_mixed_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9525624Z test_device_map_gpu_mixed_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9526072Z test_device_map_gpu_mixed_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9526566Z test_device_map_gpu_mixed_self_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9527107Z test_device_map_gpu_mixed_self_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9527601Z test_device_map_gpu_mixed_self_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9528060Z test_device_map_gpu_mixed_self_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9528547Z test_device_map_gpu_mixed_self_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9529119Z test_device_map_gpu_mixed_self_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9529509Z test_device_map_gpu_mixed_self_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9529972Z test_device_map_gpu_mixed_self_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9530474Z test_device_map_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9530996Z test_device_map_gpu_non_default_to_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9531497Z test_device_map_gpu_to_cpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9532022Z test_device_map_gpu_to_cpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9532606Z test_device_maps_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9533057Z test_device_maps_in_options (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9533467Z test_device_maps_invalid_max_local_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9533997Z test_device_maps_invalid_max_remote_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9534629Z test_device_maps_invalid_min_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9535126Z test_device_maps_many_to_one (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9535631Z test_device_maps_missing_config (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9536132Z test_device_maps_missing_config_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9536792Z test_device_maps_missing_config_not_timeout (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9537818Z test_device_maps_missing_config_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9538240Z test_device_maps_missing_config_remote_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9538788Z test_device_maps_missing_config_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9539302Z test_device_maps_missing_config_response_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9539825Z test_device_maps_multi_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9540313Z test_device_maps_multi_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9540891Z test_device_maps_one_to_many (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9541351Z test_device_maps_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9541920Z test_device_maps_return_to_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9542355Z test_device_maps_return_to_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9542868Z test_device_maps_wrong_worker_name (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9543332Z test_device_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9543803Z test_devices_option_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9544339Z test_devices_option_mismatch_reverse (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9544843Z test_owner_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9545383Z test_owner_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9546022Z test_owner_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9546535Z test_owner_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9547051Z test_rref_as_arg_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9547559Z test_rref_as_arg_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9548075Z test_rref_as_arg_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9548561Z test_rref_as_arg_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9549056Z test_rref_as_arg_synchronization5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9549578Z test_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9550104Z test_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9550611Z test_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9551125Z test_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9551637Z test_rref_to_here_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9552123Z test_rref_to_here_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9552616Z test_rref_to_here_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9553145Z test_rref_to_here_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9553659Z test_rref_with_unpickleable_attributes (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9554231Z test_tensor_view_as_return_value (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:55:36.9555177Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_device_maps_backward_pass>, <__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_dist_autograd_sync_streams>, <__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_gradients_synchronizations>]>
2022-11-23T02:55:36.9556083Z test_device_maps_backward_pass (__main__.TensorPipeTensorPipeCudaDistAutogradTest)
2022-11-23T02:55:36.9556631Z test_dist_autograd_sync_streams (__main__.TensorPipeTensorPipeCudaDistAutogradTest)
2022-11-23T02:55:36.9557150Z test_gradients_synchronizations (__main__.TensorPipeTensorPipeCudaDistAutogradTest)
2022-11-23T02:55:36.9558005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:36.9558472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:36.9559057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:36.9559596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:36.9560081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpufaw300h
2022-11-23T02:55:36.9560620Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpufaw300h/_remote_module_non_scriptable.py
2022-11-23T02:55:36.9560982Z 
2022-11-23T02:55:36.9561194Z Running tests...
2022-11-23T02:55:36.9561506Z ----------------------------------------------------------------------
2022-11-23T02:55:36.9562082Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:36.9562666Z   test_ddp_dist_autograd_local_vs_remote_gpu (__main__.TensorPipeCudaDdpComparisonTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:36.9563209Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72066
2022-11-23T02:55:36.9563666Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72067
2022-11-23T02:55:36.9564125Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 72068
2022-11-23T02:55:36.9564598Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 72069
2022-11-23T02:55:36.9565194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:36.9565654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:36.9566241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:36.9566702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:36.9567355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:36.9567756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:36.9568346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:36.9568798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:36.9569382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:36.9569826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:36.9570430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:36.9570871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:36.9571447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:36.9571970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:36.9572523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:36.9573023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:36.9573498Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1raa60z9
2022-11-23T02:55:36.9574055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1raa60z9/_remote_module_non_scriptable.py
2022-11-23T02:55:36.9574575Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqz8kkhb_
2022-11-23T02:55:36.9575122Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqz8kkhb_/_remote_module_non_scriptable.py
2022-11-23T02:55:36.9575670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzglwx7w7
2022-11-23T02:55:36.9576221Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzglwx7w7/_remote_module_non_scriptable.py
2022-11-23T02:55:36.9577270Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd4xb_apc
2022-11-23T02:55:36.9577898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd4xb_apc/_remote_module_non_scriptable.py
2022-11-23T02:55:36.9578408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:36.9578988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:36.9579460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:36.9580063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:36.9580565Z fi_getinfo: -61
2022-11-23T02:55:36.9581029Z fi_getinfo: -61
2022-11-23T02:55:36.9581321Z fi_getinfo: -61
2022-11-23T02:55:36.9581655Z fi_getinfo: -61
2022-11-23T02:55:36.9582175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:55:36.9582688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:55:36.9586055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:55:36.9586815Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:55:36.9587445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:55:36.9588114Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:55:36.9588935Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:55:36.9589702Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:55:36.9618000Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:55:36.9618606Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:55:36.9619093Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:55:36.9619594Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:55:36.9620099Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:55:36.9620591Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:55:36.9621050Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:55:36.9621541Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:55:36.9621897Z ok (7.467s)
2022-11-23T02:55:36.9622194Z 
2022-11-23T02:55:36.9622505Z ----------------------------------------------------------------------
2022-11-23T02:55:36.9622870Z Ran 1 test in 7.468s
2022-11-23T02:55:36.9623037Z 
2022-11-23T02:55:36.9623130Z OK
2022-11-23T02:55:36.9623271Z 
2022-11-23T02:55:36.9623379Z Generating XML reports...
2022-11-23T02:55:36.9624086Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDdpComparisonTest-20221123023307.xml
2022-11-23T02:55:36.9624884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:36.9625351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:36.9625919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:36.9626401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:36.9626886Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn48iup0u
2022-11-23T02:55:36.9627436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn48iup0u/_remote_module_non_scriptable.py
2022-11-23T02:55:36.9627812Z 
2022-11-23T02:55:36.9627928Z Running tests...
2022-11-23T02:55:36.9628344Z ----------------------------------------------------------------------
2022-11-23T02:55:36.9628925Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:36.9629479Z   test_gpu_simple (__main__.TensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:36.9629980Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72733
2022-11-23T02:55:36.9630449Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72734
2022-11-23T02:55:36.9630912Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 72735
2022-11-23T02:55:36.9631348Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 72736
2022-11-23T02:55:36.9631969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:36.9632438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:36.9633003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:36.9633508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:36.9634083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:36.9634544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:36.9635107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:36.9635590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:36.9636179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:36.9636644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:36.9637327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:36.9637682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:36.9638268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:36.9638701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:36.9639279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:36.9639753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:36.9640288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb_2huvyp
2022-11-23T02:55:36.9640832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb_2huvyp/_remote_module_non_scriptable.py
2022-11-23T02:55:36.9641384Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwcqtn48_
2022-11-23T02:55:36.9641932Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwcqtn48_/_remote_module_non_scriptable.py
2022-11-23T02:55:36.9642454Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvbonvs86
2022-11-23T02:55:36.9643003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvbonvs86/_remote_module_non_scriptable.py
2022-11-23T02:55:36.9643550Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgs4hczbt
2022-11-23T02:55:36.9644094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgs4hczbt/_remote_module_non_scriptable.py
2022-11-23T02:55:36.9644598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:36.9645084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:36.9645623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:36.9646098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:36.9646484Z fi_getinfo: -61
2022-11-23T02:55:36.9646772Z fi_getinfo: -61
2022-11-23T02:55:36.9647058Z fi_getinfo: -61
2022-11-23T02:55:36.9647318Z fi_getinfo: -61
2022-11-23T02:55:36.9647651Z ok (6.838s)
2022-11-23T02:55:36.9647724Z 
2022-11-23T02:55:36.9647993Z ----------------------------------------------------------------------
2022-11-23T02:55:36.9648310Z Ran 1 test in 6.838s
2022-11-23T02:55:36.9648478Z 
2022-11-23T02:55:36.9648579Z OK
2022-11-23T02:55:36.9648720Z 
2022-11-23T02:55:36.9648848Z Generating XML reports...
2022-11-23T02:55:36.9649511Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221123023317.xml
2022-11-23T02:55:36.9650299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:36.9650755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:36.9651332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:36.9651780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:36.9652245Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5x4x0q65
2022-11-23T02:55:36.9652783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5x4x0q65/_remote_module_non_scriptable.py
2022-11-23T02:55:36.9653085Z 
2022-11-23T02:55:36.9653256Z Running tests...
2022-11-23T02:55:36.9653588Z ----------------------------------------------------------------------
2022-11-23T02:55:36.9654166Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:36.9654738Z   test_gpu_to_cpu_continuation (__main__.TensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:36.9655232Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73388
2022-11-23T02:55:36.9655684Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73389
2022-11-23T02:55:36.9656133Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 73390
2022-11-23T02:55:36.9656580Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 73391
2022-11-23T02:55:36.9657678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:36.9658135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:36.9658801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:36.9659332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:36.9659854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:36.9660303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:36.9660932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:36.9661510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:36.9661962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:36.9662407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:36.9662960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:36.9663427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:36.9664076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:36.9664640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:36.9665069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:36.9665532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:36.9666003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp52p8r07d
2022-11-23T02:55:36.9666548Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp52p8r07d/_remote_module_non_scriptable.py
2022-11-23T02:55:36.9667068Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpld8c3fhn
2022-11-23T02:55:36.9667675Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpld8c3fhn/_remote_module_non_scriptable.py
2022-11-23T02:55:36.9668145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzuwxq90h
2022-11-23T02:55:36.9668665Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzuwxq90h/_remote_module_non_scriptable.py
2022-11-23T02:55:36.9669197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbytnpr6i
2022-11-23T02:55:36.9669733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbytnpr6i/_remote_module_non_scriptable.py
2022-11-23T02:55:36.9670241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:36.9670699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:36.9671162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:36.9671638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:36.9672020Z fi_getinfo: -61
2022-11-23T02:55:36.9672296Z fi_getinfo: -61
2022-11-23T02:55:36.9672565Z fi_getinfo: -61
2022-11-23T02:55:36.9672821Z fi_getinfo: -61
2022-11-23T02:55:36.9673054Z ok (6.814s)
2022-11-23T02:55:36.9673203Z 
2022-11-23T02:55:36.9673472Z ----------------------------------------------------------------------
2022-11-23T02:55:36.9673799Z Ran 1 test in 6.814s
2022-11-23T02:55:36.9673943Z 
2022-11-23T02:55:36.9674036Z OK
2022-11-23T02:55:36.9674171Z 
2022-11-23T02:55:36.9674308Z Generating XML reports...
2022-11-23T02:55:36.9674973Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221123023327.xml
2022-11-23T02:55:36.9675730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:36.9676241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:36.9676827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:36.9677303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:36.9677905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmey3yqke
2022-11-23T02:55:36.9678620Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmey3yqke/_remote_module_non_scriptable.py
2022-11-23T02:55:36.9678922Z 
2022-11-23T02:55:36.9679033Z Running tests...
2022-11-23T02:55:36.9679430Z ----------------------------------------------------------------------
2022-11-23T02:55:36.9680004Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:36.9680581Z   test_gpu_to_cpu_continuation_gpu_root (__main__.TensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:36.9681162Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74043
2022-11-23T02:55:36.9681604Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74044
2022-11-23T02:55:36.9682115Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 74045
2022-11-23T02:55:36.9682562Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 74046
2022-11-23T02:55:36.9683160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:36.9683615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:36.9684185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:36.9684650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:36.9685403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:36.9685805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:36.9686358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:36.9686802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:36.9687342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:36.9687938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:36.9688499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:36.9688938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:36.9689509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:36.9689952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:36.9690525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:36.9691110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:36.9691562Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu58ssc8o
2022-11-23T02:55:36.9692085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu58ssc8o/_remote_module_non_scriptable.py
2022-11-23T02:55:36.9692606Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg1s5sb1v
2022-11-23T02:55:36.9693106Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg1s5sb1v/_remote_module_non_scriptable.py
2022-11-23T02:55:36.9693793Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplneomkkv
2022-11-23T02:55:36.9694382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplneomkkv/_remote_module_non_scriptable.py
2022-11-23T02:55:36.9694905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphx11aa0v
2022-11-23T02:55:36.9695446Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphx11aa0v/_remote_module_non_scriptable.py
2022-11-23T02:55:36.9695952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:36.9696423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:36.9697142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:36.9697556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:36.9697953Z fi_getinfo: -61
2022-11-23T02:55:36.9698214Z fi_getinfo: -61
2022-11-23T02:55:36.9698484Z fi_getinfo: -61
2022-11-23T02:55:36.9698753Z fi_getinfo: -61
2022-11-23T02:55:36.9699069Z ok (7.039s)
2022-11-23T02:55:36.9699117Z 
2022-11-23T02:55:36.9699387Z ----------------------------------------------------------------------
2022-11-23T02:55:36.9699797Z Ran 1 test in 7.039s
2022-11-23T02:55:36.9699959Z 
2022-11-23T02:55:36.9700054Z OK
2022-11-23T02:55:36.9700172Z 
2022-11-23T02:55:36.9700455Z Generating XML reports...
2022-11-23T02:55:36.9701121Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221123023337.xml
2022-11-23T02:55:36.9701870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:36.9702288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:36.9702842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:36.9703295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:36.9703759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp49x57jgx
2022-11-23T02:55:36.9704267Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp49x57jgx/_remote_module_non_scriptable.py
2022-11-23T02:55:36.9704563Z 
2022-11-23T02:55:36.9704668Z Running tests...
2022-11-23T02:55:36.9705057Z ----------------------------------------------------------------------
2022-11-23T02:55:36.9705587Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:36.9706138Z   test_input_moved_to_cuda_device (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:36.9706638Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74698
2022-11-23T02:55:36.9707075Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74699
2022-11-23T02:55:36.9707646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:36.9708257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:36.9708839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:36.9709287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:36.9709867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:36.9710307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:36.9710875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:36.9711465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:36.9712090Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9p3xl90w
2022-11-23T02:55:36.9712718Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9p3xl90w/_remote_module_non_scriptable.py
2022-11-23T02:55:36.9713267Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx7ap3ak2
2022-11-23T02:55:36.9713789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx7ap3ak2/_remote_module_non_scriptable.py
2022-11-23T02:55:36.9714296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:36.9714900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:36.9715261Z fi_getinfo: -61
2022-11-23T02:55:36.9715515Z fi_getinfo: -61
2022-11-23T02:55:36.9715726Z ok (6.204s)
2022-11-23T02:55:36.9715860Z 
2022-11-23T02:55:36.9716100Z ----------------------------------------------------------------------
2022-11-23T02:55:36.9716398Z Ran 1 test in 6.204s
2022-11-23T02:55:36.9716551Z 
2022-11-23T02:55:36.9716641Z OK
2022-11-23T02:55:36.9716771Z 
2022-11-23T02:55:36.9716891Z Generating XML reports...
2022-11-23T02:55:36.9717532Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221123023346.xml
2022-11-23T02:55:36.9718336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:36.9718775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:36.9719312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:36.9719953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:36.9720417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ycyyfnq
2022-11-23T02:55:36.9720956Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ycyyfnq/_remote_module_non_scriptable.py
2022-11-23T02:55:36.9721296Z 
2022-11-23T02:55:36.9721348Z Running tests...
2022-11-23T02:55:36.9721761Z ----------------------------------------------------------------------
2022-11-23T02:55:36.9722340Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:36.9723251Z   test_input_moved_to_cuda_device_script (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:36.9723758Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74984
2022-11-23T02:55:36.9724210Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74985
2022-11-23T02:55:36.9724819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:36.9725251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:36.9725817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:36.9726285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:36.9726864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:36.9727293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:36.9727985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:36.9728474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:36.9728907Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkni08a4t
2022-11-23T02:55:36.9729434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkni08a4t/_remote_module_non_scriptable.py
2022-11-23T02:55:36.9729953Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpny29365r
2022-11-23T02:55:36.9730513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpny29365r/_remote_module_non_scriptable.py
2022-11-23T02:55:36.9731166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:36.9731648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:36.9732046Z fi_getinfo: -61
2022-11-23T02:55:36.9732304Z fi_getinfo: -61
2022-11-23T02:55:36.9732808Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkni08a4t/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-11-23T02:55:36.9733539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpny29365r/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-11-23T02:55:36.9734276Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmpny29365r/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-11-23T02:55:36.9734734Z ok (6.435s)
2022-11-23T02:55:36.9734885Z 
2022-11-23T02:55:36.9735158Z ----------------------------------------------------------------------
2022-11-23T02:55:36.9735487Z Ran 1 test in 6.435s
2022-11-23T02:55:36.9735702Z 
2022-11-23T02:55:36.9735798Z OK
2022-11-23T02:55:36.9735913Z 
2022-11-23T02:55:36.9736040Z Generating XML reports...
2022-11-23T02:55:36.9736895Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221123023355.xml
2022-11-23T02:55:36.9737680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:36.9738112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:36.9738688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:36.9739156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:36.9739622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3qcjykvw
2022-11-23T02:55:36.9740147Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3qcjykvw/_remote_module_non_scriptable.py
2022-11-23T02:55:36.9740454Z 
2022-11-23T02:55:36.9740560Z Running tests...
2022-11-23T02:55:36.9740966Z ----------------------------------------------------------------------
2022-11-23T02:55:36.9741540Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:36.9742188Z   test_invalid_devices (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:36.9742838Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75302
2022-11-23T02:55:36.9743447Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75303
2022-11-23T02:55:36.9744048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:36.9744499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:36.9745078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:36.9745556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:36.9746264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:36.9746875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:36.9747440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:36.9748004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:36.9748362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsrb2w1ob
2022-11-23T02:55:36.9748986Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsrb2w1ob/_remote_module_non_scriptable.py
2022-11-23T02:55:36.9749534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0nnigxd4
2022-11-23T02:55:36.9750057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0nnigxd4/_remote_module_non_scriptable.py
2022-11-23T02:55:36.9750565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:36.9751035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:36.9751414Z fi_getinfo: -61
2022-11-23T02:55:36.9751686Z fi_getinfo: -61
2022-11-23T02:55:36.9751954Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:55:36.9770226Z RuntimeError('CUDA error: invalid device ordinal\nCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nException raised from c10_cuda_check_implementation at /var/lib/jenkins/workspace/c10/cuda/CUDAException.cpp:31 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f7d3d7c259b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f7d3d7bddfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #2: c10::cuda::c10_cuda_check_implementation(char const*, char const*, int, bool) + 0x42e (0x7f7d3da4806e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)\nframe #3: <unknown function> + 0x17c9d (0x7f7d3da20c9d in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)\nframe #4: <unknown function> + 0xdf3ecd (0x7f7d3ea60ecd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #5: <unknown function> + 0x29c7ab5 (0x7f7d40634ab5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #6: <unknown function> + 0x29c7c5b (0x7f7d40634c5b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #7: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x1e3 (0x7f7d4a837cb3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #8: <unknown function> + 0x20c43b5 (0x7f7d4ab593b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: at::_ops::empty_strided::call(c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x168 (0x7f7d4a872e58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: <unknown function> + 0x12691af (0x7f7d49cfe1af in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::native::_to_copy(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x1321 (0x7f7d4a094641 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: <unknown function> + 0x22a6e23 (0x7f7d4ad3be23 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #13: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f7d4a5580e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #14: <unknown function> + 0x20c8908 (0x7f7d4ab5d908 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #15: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f7d4a5580e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #16: <unknown function> + 0x3443e11 (0x7f7d4bed8e11 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #17: <unknown function> + 0x34443bb (0x7f7d4bed93bb in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #18: at::_ops::_to_copy::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x201 (0x7f7d4a5b1de1 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #19: at::native::to(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x13e (0x7f7d4a08c05e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #20: <unknown function> + 0x2471e09 (0x7f7d4af06e09 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #21: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x215 (0x7f7d4a7233f5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #22: <unknown function> + 0x36467f (0x7f7d557d867f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #23: <unknown function> + 0x364b3c (0x7f7d557d8b3c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #24: <unknown function> + 0x1ddc68 (0x556a87974c68 in /opt/conda/bin/python)\nframe #25: <unknown function> + 0x1049f3 (0x556a8789b9f3 in /opt/conda/bin/python)\nframe #26: <unknown function> + 0x18f742 (0x556a87926742 in /opt/conda/bin/python)\nframe #27: <unknown function> + 0x104425 (0x556a8789b425 in /opt/conda/bin/python)\nframe #28: <unknown function> + 0x18f742 (0x556a87926742 in /opt/conda/bin/python)\nframe #29: <unknown function> + 0x18fc9b (0x556a87926c9b in /opt/conda/bin/python)\nframe #30: <unknown function> + 0x1052a5 (0x556a8789c2a5 in /opt/conda/bin/python)\nframe #31: <unknown function> + 0x18f742 (0x556a87926742 in /opt/conda/bin/python)\nframe #32: <unknown function> + 0x18fc9b (0x556a87926c9b in /opt/conda/bin/python)\nframe #33: <unknown function> + 0x1052a5 (0x556a8789c2a5 in /opt/conda/bin/python)\nframe #34: <unknown function> + 0x18f742 (0x556a87926742 in /opt/conda/bin/python)\nframe #35: _PyObject_Call + 0x20a (0x556a878defaa in /opt/conda/bin/python)\nframe #36: _PyEval_EvalFrameDefault + 0x26e4 (0x556a8797a774 in /opt/conda/bin/python)\nframe #37: <unknown function> + 0x18f742 (0x556a87926742 in /opt/conda/bin/python)\nframe #38: _PyObject_Call + 0x20a (0x556a878defaa in /opt/conda/bin/python)\nframe #39: <unknown function> + 0xaa8dba (0x7f7d55f1cdba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #40: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f7d55f1affd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #41: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f7d55f1e2d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #42: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x83 (0x7f7d55f1e9a3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #43: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x194 (0x7f7d4d4c0654 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #44: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f7d55f1e0c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #45: <unknown function> + 0x4a24a53 (0x7f7d4d4b9a53 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #46: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f7d4d4ba5e8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #47: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f7d4d4b48e7 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #48: <unknown function> + 0x4a545d2 (0x7f7d4d4e95d2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #49: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f7d3d7b090b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #50: <unknown function> + 0xdbbf4 (0x7f7d6d616bf4 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #51: <unknown function> + 0x76db (0x7f7d8dc6a6db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #52: clone + 0x3f (0x7f7d8d99361f in /lib/x86_64-linux-gnu/libc.so.6)\n')
2022-11-23T02:55:36.9780873Z Traceback (most recent call last):
2022-11-23T02:55:36.9781426Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:55:36.9781890Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:55:36.9782477Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/nn/api/remote_module.py", line 92, in _create_module
2022-11-23T02:55:36.9782850Z     module.to(device)
2022-11-23T02:55:36.9783309Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1120, in to
2022-11-23T02:55:36.9783686Z     return self._apply(convert)
2022-11-23T02:55:36.9784152Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 795, in _apply
2022-11-23T02:55:36.9784525Z     param_applied = fn(param)
2022-11-23T02:55:36.9785011Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1118, in convert
2022-11-23T02:55:36.9785635Z     return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
2022-11-23T02:55:36.9786028Z RuntimeError: CUDA error: invalid device ordinal
2022-11-23T02:55:36.9786447Z CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
2022-11-23T02:55:36.9787002Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
2022-11-23T02:55:36.9787352Z Exception raised from c10_cuda_check_implementation at /var/lib/jenkins/workspace/c10/cuda/CUDAException.cpp:31 (most recent call first):
2022-11-23T02:55:36.9788369Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f7d3d7c259b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:55:36.9789348Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f7d3d7bddfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:55:36.9790244Z frame #2: c10::cuda::c10_cuda_check_implementation(char const*, char const*, int, bool) + 0x42e (0x7f7d3da4806e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
2022-11-23T02:55:36.9791187Z frame #3: <unknown function> + 0x17c9d (0x7f7d3da20c9d in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
2022-11-23T02:55:36.9791813Z frame #4: <unknown function> + 0xdf3ecd (0x7f7d3ea60ecd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:55:36.9792449Z frame #5: <unknown function> + 0x29c7ab5 (0x7f7d40634ab5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:55:36.9793066Z frame #6: <unknown function> + 0x29c7c5b (0x7f7d40634c5b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:55:36.9794212Z frame #7: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x1e3 (0x7f7d4a837cb3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9795034Z frame #8: <unknown function> + 0x20c43b5 (0x7f7d4ab593b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9795992Z frame #9: at::_ops::empty_strided::call(c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x168 (0x7f7d4a872e58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9796878Z frame #10: <unknown function> + 0x12691af (0x7f7d49cfe1af in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9797957Z frame #11: at::native::_to_copy(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x1321 (0x7f7d4a094641 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9798753Z frame #12: <unknown function> + 0x22a6e23 (0x7f7d4ad3be23 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9799832Z frame #13: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f7d4a5580e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9800638Z frame #14: <unknown function> + 0x20c8908 (0x7f7d4ab5d908 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9801504Z frame #15: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f7d4a5580e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9802326Z frame #16: <unknown function> + 0x3443e11 (0x7f7d4bed8e11 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9802946Z frame #17: <unknown function> + 0x34443bb (0x7f7d4bed93bb in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9803872Z frame #18: at::_ops::_to_copy::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x201 (0x7f7d4a5b1de1 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9804932Z frame #19: at::native::to(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x13e (0x7f7d4a08c05e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9805705Z frame #20: <unknown function> + 0x2471e09 (0x7f7d4af06e09 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9806728Z frame #21: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x215 (0x7f7d4a7233f5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9807544Z frame #22: <unknown function> + 0x36467f (0x7f7d557d867f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:36.9808175Z frame #23: <unknown function> + 0x364b3c (0x7f7d557d8b3c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:36.9808783Z frame #24: <unknown function> + 0x1ddc68 (0x556a87974c68 in /opt/conda/bin/python)
2022-11-23T02:55:36.9809186Z frame #25: <unknown function> + 0x1049f3 (0x556a8789b9f3 in /opt/conda/bin/python)
2022-11-23T02:55:36.9809581Z frame #26: <unknown function> + 0x18f742 (0x556a87926742 in /opt/conda/bin/python)
2022-11-23T02:55:36.9809948Z frame #27: <unknown function> + 0x104425 (0x556a8789b425 in /opt/conda/bin/python)
2022-11-23T02:55:36.9810341Z frame #28: <unknown function> + 0x18f742 (0x556a87926742 in /opt/conda/bin/python)
2022-11-23T02:55:36.9810735Z frame #29: <unknown function> + 0x18fc9b (0x556a87926c9b in /opt/conda/bin/python)
2022-11-23T02:55:36.9811226Z frame #30: <unknown function> + 0x1052a5 (0x556a8789c2a5 in /opt/conda/bin/python)
2022-11-23T02:55:36.9811733Z frame #31: <unknown function> + 0x18f742 (0x556a87926742 in /opt/conda/bin/python)
2022-11-23T02:55:36.9812271Z frame #32: <unknown function> + 0x18fc9b (0x556a87926c9b in /opt/conda/bin/python)
2022-11-23T02:55:36.9812663Z frame #33: <unknown function> + 0x1052a5 (0x556a8789c2a5 in /opt/conda/bin/python)
2022-11-23T02:55:36.9813054Z frame #34: <unknown function> + 0x18f742 (0x556a87926742 in /opt/conda/bin/python)
2022-11-23T02:55:36.9813426Z frame #35: _PyObject_Call + 0x20a (0x556a878defaa in /opt/conda/bin/python)
2022-11-23T02:55:36.9813844Z frame #36: _PyEval_EvalFrameDefault + 0x26e4 (0x556a8797a774 in /opt/conda/bin/python)
2022-11-23T02:55:36.9814258Z frame #37: <unknown function> + 0x18f742 (0x556a87926742 in /opt/conda/bin/python)
2022-11-23T02:55:36.9814627Z frame #38: _PyObject_Call + 0x20a (0x556a878defaa in /opt/conda/bin/python)
2022-11-23T02:55:36.9815386Z frame #39: <unknown function> + 0xaa8dba (0x7f7d55f1cdba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:36.9816157Z frame #40: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f7d55f1affd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:36.9817518Z frame #41: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f7d55f1e2d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:36.9818668Z frame #42: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x83 (0x7f7d55f1e9a3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:36.9819879Z frame #43: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x194 (0x7f7d4d4c0654 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9821157Z frame #44: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f7d55f1e0c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:36.9822050Z frame #45: <unknown function> + 0x4a24a53 (0x7f7d4d4b9a53 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9823085Z frame #46: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f7d4d4ba5e8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9824307Z frame #47: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f7d4d4b48e7 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9825072Z frame #48: <unknown function> + 0x4a545d2 (0x7f7d4d4e95d2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9825895Z frame #49: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f7d3d7b090b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:55:36.9826404Z frame #50: <unknown function> + 0xdbbf4 (0x7f7d6d616bf4 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-11-23T02:55:36.9826969Z frame #51: <unknown function> + 0x76db (0x7f7d8dc6a6db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-11-23T02:55:36.9827476Z frame #52: clone + 0x3f (0x7f7d8d99361f in /lib/x86_64-linux-gnu/libc.so.6)
2022-11-23T02:55:36.9827758Z 
2022-11-23T02:55:36.9827777Z 
2022-11-23T02:55:36.9827912Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:55:36.9865821Z RuntimeError('On WorkerInfo(id=1, name=worker1):\nRuntimeError(\'CUDA error: invalid device ordinal\nCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nException raised from c10_cuda_check_implementation at /var/lib/jenkins/workspace/c10/cuda/CUDAException.cpp:31 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f7d3d7c259b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f7d3d7bddfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #2: c10::cuda::c10_cuda_check_implementation(char const*, char const*, int, bool) + 0x42e (0x7f7d3da4806e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)\nframe #3: <unknown function> + 0x17c9d (0x7f7d3da20c9d in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)\nframe #4: <unknown function> + 0xdf3ecd (0x7f7d3ea60ecd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #5: <unknown function> + 0x29c7ab5 (0x7f7d40634ab5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #6: <unknown function> + 0x29c7c5b (0x7f7d40634c5b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #7: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x1e3 (0x7f7d4a837cb3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #8: <unknown function> + 0x20c43b5 (0x7f7d4ab593b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: at::_ops::empty_strided::call(c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x168 (0x7f7d4a872e58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: <unknown function> + 0x12691af (0x7f7d49cfe1af in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::native::_to_copy(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x1321 (0x7f7d4a094641 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: <unknown function> + 0x22a6e23 (0x7f7d4ad3be23 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #13: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f7d4a5580e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #14: <unknown function> + 0x20c8908 (0x7f7d4ab5d908 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #15: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f7d4a5580e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #16: <unknown function> + 0x3443e11 (0x7f7d4bed8e11 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #17: <unknown function> + 0x34443bb (0x7f7d4bed93bb in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #18: at::_ops::_to_copy::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x201 (0x7f7d4a5b1de1 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #19: at::native::to(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x13e (0x7f7d4a08c05e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #20: <unknown function> + 0x2471e09 (0x7f7d4af06e09 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #21: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x215 (0x7f7d4a7233f5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #22: <unknown function> + 0x36467f (0x7f7d557d867f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #23: <unknown function> + 0x364b3c (0x7f7d557d8b3c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #24: <unknown function> + 0x1ddc68 (0x556a87974c68 in /opt/conda/bin/python)\nframe #25: <unknown function> + 0x1049f3 (0x556a8789b9f3 in /opt/conda/bin/python)\nframe #26: <unknown function> + 0x18f742 (0x556a87926742 in /opt/conda/bin/python)\nframe #27: <unknown function> + 0x104425 (0x556a8789b425 in /opt/conda/bin/python)\nframe #28: <unknown function> + 0x18f742 (0x556a87926742 in /opt/conda/bin/python)\nframe #29: <unknown function> + 0x18fc9b (0x556a87926c9b in /opt/conda/bin/python)\nframe #30: <unknown function> + 0x1052a5 (0x556a8789c2a5 in /opt/conda/bin/python)\nframe #31: <unknown function> + 0x18f742 (0x556a87926742 in /opt/conda/bin/python)\nframe #32: <unknown function> + 0x18fc9b (0x556a87926c9b in /opt/conda/bin/python)\nframe #33: <unknown function> + 0x1052a5 (0x556a8789c2a5 in /opt/conda/bin/python)\nframe #34: <unknown function> + 0x18f742 (0x556a87926742 in /opt/conda/bin/python)\nframe #35: _PyObject_Call + 0x20a (0x556a878defaa in /opt/conda/bin/python)\nframe #36: _PyEval_EvalFrameDefault + 0x26e4 (0x556a8797a774 in /opt/conda/bin/python)\nframe #37: <unknown function> + 0x18f742 (0x556a87926742 in /opt/conda/bin/python)\nframe #38: _PyObject_Call + 0x20a (0x556a878defaa in /opt/conda/bin/python)\nframe #39: <unknown function> + 0xaa8dba (0x7f7d55f1cdba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #40: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f7d55f1affd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #41: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f7d55f1e2d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #42: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x83 (0x7f7d55f1e9a3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #43: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x194 (0x7f7d4d4c0654 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #44: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f7d55f1e0c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #45: <unknown function> + 0x4a24a53 (0x7f7d4d4b9a53 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #46: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f7d4d4ba5e8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #47: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f7d4d4b48e7 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #48: <unknown function> + 0x4a545d2 (0x7f7d4d4e95d2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #49: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f7d3d7b090b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #50: <unknown function> + 0xdbbf4 (0x7f7d6d616bf4 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #51: <unknown function> + 0x76db (0x7f7d8dc6a6db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #52: clone + 0x3f (0x7f7d8d99361f in /lib/x86_64-linux-gnu/libc.so.6)\n\')\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.10/site-packages/torch/distributed/nn/api/remote_module.py", line 92, in _create_module\n    module.to(device)\n  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1120, in to\n    return self._apply(convert)\n  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 795, in _apply\n    param_applied = fn(param)\n  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1118, in convert\n    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)\nRuntimeError: CUDA error: invalid device ordinal\nCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nException raised from c10_cuda_check_implementation at /var/lib/jenkins/workspace/c10/cuda/CUDAException.cpp:31 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f7d3d7c259b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f7d3d7bddfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #2: c10::cuda::c10_cuda_check_implementation(char const*, char const*, int, bool) + 0x42e (0x7f7d3da4806e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)\nframe #3: <unknown function> + 0x17c9d (0x7f7d3da20c9d in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)\nframe #4: <unknown function> + 0xdf3ecd (0x7f7d3ea60ecd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #5: <unknown function> + 0x29c7ab5 (0x7f7d40634ab5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #6: <unknown function> + 0x29c7c5b (0x7f7d40634c5b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #7: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x1e3 (0x7f7d4a837cb3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #8: <unknown function> + 0x20c43b5 (0x7f7d4ab593b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: at::_ops::empty_strided::call(c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x168 (0x7f7d4a872e58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: <unknown function> + 0x12691af (0x7f7d49cfe1af in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::native::_to_copy(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x1321 (0x7f7d4a094641 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: <unknown function> + 0x22a6e23 (0x7f7d4ad3be23 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #13: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f7d4a5580e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #14: <unknown function> + 0x20c8908 (0x7f7d4ab5d908 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #15: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f7d4a5580e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #16: <unknown function> + 0x3443e11 (0x7f7d4bed8e11 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #17: <unknown function> + 0x34443bb (0x7f7d4bed93bb in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #18: at::_ops::_to_copy::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x201 (0x7f7d4a5b1de1 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #19: at::native::to(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x13e (0x7f7d4a08c05e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #20: <unknown function> + 0x2471e09 (0x7f7d4af06e09 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #21: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x215 (0x7f7d4a7233f5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #22: <unknown function> + 0x36467f (0x7f7d557d867f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #23: <unknown function> + 0x364b3c (0x7f7d557d8b3c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #24: <unknown function> + 0x1ddc68 (0x556a87974c68 in /opt/conda/bin/python)\nframe #25: <unknown function> + 0x1049f3 (0x556a8789b9f3 in /opt/conda/bin/python)\nframe #26: <unknown function> + 0x18f742 (0x556a87926742 in /opt/conda/bin/python)\nframe #27: <unknown function> + 0x104425 (0x556a8789b425 in /opt/conda/bin/python)\nframe #28: <unknown function> + 0x18f742 (0x556a87926742 in /opt/conda/bin/python)\nframe #29: <unknown function> + 0x18fc9b (0x556a87926c9b in /opt/conda/bin/python)\nframe #30: <unknown function> + 0x1052a5 (0x556a8789c2a5 in /opt/conda/bin/python)\nframe #31: <unknown function> + 0x18f742 (0x556a87926742 in /opt/conda/bin/python)\nframe #32: <unknown function> + 0x18fc9b (0x556a87926c9b in /opt/conda/bin/python)\nframe #33: <unknown function> + 0x1052a5 (0x556a8789c2a5 in /opt/conda/bin/python)\nframe #34: <unknown function> + 0x18f742 (0x556a87926742 in /opt/conda/bin/python)\nframe #35: _PyObject_Call + 0x20a (0x556a878defaa in /opt/conda/bin/python)\nframe #36: _PyEval_EvalFrameDefault + 0x26e4 (0x556a8797a774 in /opt/conda/bin/python)\nframe #37: <unknown function> + 0x18f742 (0x556a87926742 in /opt/conda/bin/python)\nframe #38: _PyObject_Call + 0x20a (0x556a878defaa in /opt/conda/bin/python)\nframe #39: <unknown function> + 0xaa8dba (0x7f7d55f1cdba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #40: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f7d55f1affd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #41: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f7d55f1e2d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #42: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x83 (0x7f7d55f1e9a3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #43: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x194 (0x7f7d4d4c0654 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #44: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f7d55f1e0c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #45: <unknown function> + 0x4a24a53 (0x7f7d4d4b9a53 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #46: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f7d4d4ba5e8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #47: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f7d4d4b48e7 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #48: <unknown function> + 0x4a545d2 (0x7f7d4d4e95d2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #49: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f7d3d7b090b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #50: <unknown function> + 0xdbbf4 (0x7f7d6d616bf4 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #51: <unknown function> + 0x76db (0x7f7d8dc6a6db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #52: clone + 0x3f (0x7f7d8d99361f in /lib/x86_64-linux-gnu/libc.so.6)\n\n')
2022-11-23T02:55:36.9888822Z Traceback (most recent call last):
2022-11-23T02:55:36.9889393Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:55:36.9889846Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:55:36.9890287Z   File "/tmp/tmp3qcjykvw/_remote_module_non_scriptable.py", line 47, in _remote_forward
2022-11-23T02:55:36.9890676Z     module = module_rref.local_value()
2022-11-23T02:55:36.9891200Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 236, in _handle_exception
2022-11-23T02:55:36.9891737Z     raise exc
2022-11-23T02:55:36.9892030Z RuntimeError: On WorkerInfo(id=1, name=worker1):
2022-11-23T02:55:36.9892435Z RuntimeError('CUDA error: invalid device ordinal
2022-11-23T02:55:36.9892859Z CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
2022-11-23T02:55:36.9893313Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
2022-11-23T02:55:36.9893807Z Exception raised from c10_cuda_check_implementation at /var/lib/jenkins/workspace/c10/cuda/CUDAException.cpp:31 (most recent call first):
2022-11-23T02:55:36.9894955Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f7d3d7c259b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:55:36.9895942Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f7d3d7bddfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:55:36.9897129Z frame #2: c10::cuda::c10_cuda_check_implementation(char const*, char const*, int, bool) + 0x42e (0x7f7d3da4806e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
2022-11-23T02:55:36.9897760Z frame #3: <unknown function> + 0x17c9d (0x7f7d3da20c9d in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
2022-11-23T02:55:36.9898422Z frame #4: <unknown function> + 0xdf3ecd (0x7f7d3ea60ecd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:55:36.9899071Z frame #5: <unknown function> + 0x29c7ab5 (0x7f7d40634ab5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:55:36.9899726Z frame #6: <unknown function> + 0x29c7c5b (0x7f7d40634c5b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:55:36.9900904Z frame #7: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x1e3 (0x7f7d4a837cb3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9901934Z frame #8: <unknown function> + 0x20c43b5 (0x7f7d4ab593b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9902906Z frame #9: at::_ops::empty_strided::call(c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x168 (0x7f7d4a872e58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9903710Z frame #10: <unknown function> + 0x12691af (0x7f7d49cfe1af in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9904830Z frame #11: at::native::_to_copy(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x1321 (0x7f7d4a094641 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9905624Z frame #12: <unknown function> + 0x22a6e23 (0x7f7d4ad3be23 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9906692Z frame #13: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f7d4a5580e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9907541Z frame #14: <unknown function> + 0x20c8908 (0x7f7d4ab5d908 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9908506Z frame #15: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f7d4a5580e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9909568Z frame #16: <unknown function> + 0x3443e11 (0x7f7d4bed8e11 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9910189Z frame #17: <unknown function> + 0x34443bb (0x7f7d4bed93bb in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9911226Z frame #18: at::_ops::_to_copy::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x201 (0x7f7d4a5b1de1 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9912668Z frame #19: at::native::to(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x13e (0x7f7d4a08c05e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9913487Z frame #20: <unknown function> + 0x2471e09 (0x7f7d4af06e09 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9914452Z frame #21: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x215 (0x7f7d4a7233f5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9915301Z frame #22: <unknown function> + 0x36467f (0x7f7d557d867f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:36.9916126Z frame #23: <unknown function> + 0x364b3c (0x7f7d557d8b3c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:36.9916590Z frame #24: <unknown function> + 0x1ddc68 (0x556a87974c68 in /opt/conda/bin/python)
2022-11-23T02:55:36.9916964Z frame #25: <unknown function> + 0x1049f3 (0x556a8789b9f3 in /opt/conda/bin/python)
2022-11-23T02:55:36.9917358Z frame #26: <unknown function> + 0x18f742 (0x556a87926742 in /opt/conda/bin/python)
2022-11-23T02:55:36.9917813Z frame #27: <unknown function> + 0x104425 (0x556a8789b425 in /opt/conda/bin/python)
2022-11-23T02:55:36.9918182Z frame #28: <unknown function> + 0x18f742 (0x556a87926742 in /opt/conda/bin/python)
2022-11-23T02:55:36.9918573Z frame #29: <unknown function> + 0x18fc9b (0x556a87926c9b in /opt/conda/bin/python)
2022-11-23T02:55:36.9918965Z frame #30: <unknown function> + 0x1052a5 (0x556a8789c2a5 in /opt/conda/bin/python)
2022-11-23T02:55:36.9919352Z frame #31: <unknown function> + 0x18f742 (0x556a87926742 in /opt/conda/bin/python)
2022-11-23T02:55:36.9919714Z frame #32: <unknown function> + 0x18fc9b (0x556a87926c9b in /opt/conda/bin/python)
2022-11-23T02:55:36.9920097Z frame #33: <unknown function> + 0x1052a5 (0x556a8789c2a5 in /opt/conda/bin/python)
2022-11-23T02:55:36.9920655Z frame #34: <unknown function> + 0x18f742 (0x556a87926742 in /opt/conda/bin/python)
2022-11-23T02:55:36.9921058Z frame #35: _PyObject_Call + 0x20a (0x556a878defaa in /opt/conda/bin/python)
2022-11-23T02:55:36.9921467Z frame #36: _PyEval_EvalFrameDefault + 0x26e4 (0x556a8797a774 in /opt/conda/bin/python)
2022-11-23T02:55:36.9921970Z frame #37: <unknown function> + 0x18f742 (0x556a87926742 in /opt/conda/bin/python)
2022-11-23T02:55:36.9922353Z frame #38: _PyObject_Call + 0x20a (0x556a878defaa in /opt/conda/bin/python)
2022-11-23T02:55:36.9922943Z frame #39: <unknown function> + 0xaa8dba (0x7f7d55f1cdba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:36.9923894Z frame #40: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f7d55f1affd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:36.9924869Z frame #41: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f7d55f1e2d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:36.9926139Z frame #42: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x83 (0x7f7d55f1e9a3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:36.9927427Z frame #43: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x194 (0x7f7d4d4c0654 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9928706Z frame #44: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f7d55f1e0c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:36.9929715Z frame #45: <unknown function> + 0x4a24a53 (0x7f7d4d4b9a53 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9930641Z frame #46: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f7d4d4ba5e8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9931852Z frame #47: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f7d4d4b48e7 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9932653Z frame #48: <unknown function> + 0x4a545d2 (0x7f7d4d4e95d2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9933354Z frame #49: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f7d3d7b090b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:55:36.9933849Z frame #50: <unknown function> + 0xdbbf4 (0x7f7d6d616bf4 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-11-23T02:55:36.9934420Z frame #51: <unknown function> + 0x76db (0x7f7d8dc6a6db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-11-23T02:55:36.9934944Z frame #52: clone + 0x3f (0x7f7d8d99361f in /lib/x86_64-linux-gnu/libc.so.6)
2022-11-23T02:55:36.9935249Z ')
2022-11-23T02:55:36.9935660Z Traceback (most recent call last):
2022-11-23T02:55:36.9936176Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:55:36.9937224Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:55:36.9937737Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/nn/api/remote_module.py", line 92, in _create_module
2022-11-23T02:55:36.9938133Z     module.to(device)
2022-11-23T02:55:36.9938609Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1120, in to
2022-11-23T02:55:36.9938967Z     return self._apply(convert)
2022-11-23T02:55:36.9939537Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 795, in _apply
2022-11-23T02:55:36.9939933Z     param_applied = fn(param)
2022-11-23T02:55:36.9940790Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1118, in convert
2022-11-23T02:55:36.9941233Z     return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
2022-11-23T02:55:36.9941646Z RuntimeError: CUDA error: invalid device ordinal
2022-11-23T02:55:36.9942110Z CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
2022-11-23T02:55:36.9942550Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
2022-11-23T02:55:36.9943053Z Exception raised from c10_cuda_check_implementation at /var/lib/jenkins/workspace/c10/cuda/CUDAException.cpp:31 (most recent call first):
2022-11-23T02:55:36.9944067Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f7d3d7c259b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:55:36.9945028Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f7d3d7bddfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:55:36.9945997Z frame #2: c10::cuda::c10_cuda_check_implementation(char const*, char const*, int, bool) + 0x42e (0x7f7d3da4806e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
2022-11-23T02:55:36.9946659Z frame #3: <unknown function> + 0x17c9d (0x7f7d3da20c9d in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
2022-11-23T02:55:36.9947476Z frame #4: <unknown function> + 0xdf3ecd (0x7f7d3ea60ecd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:55:36.9948140Z frame #5: <unknown function> + 0x29c7ab5 (0x7f7d40634ab5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:55:36.9948798Z frame #6: <unknown function> + 0x29c7c5b (0x7f7d40634c5b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:55:36.9949815Z frame #7: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x1e3 (0x7f7d4a837cb3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9950671Z frame #8: <unknown function> + 0x20c43b5 (0x7f7d4ab593b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9951629Z frame #9: at::_ops::empty_strided::call(c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x168 (0x7f7d4a872e58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9952445Z frame #10: <unknown function> + 0x12691af (0x7f7d49cfe1af in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9953404Z frame #11: at::native::_to_copy(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x1321 (0x7f7d4a094641 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9954331Z frame #12: <unknown function> + 0x22a6e23 (0x7f7d4ad3be23 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9955367Z frame #13: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f7d4a5580e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9956433Z frame #14: <unknown function> + 0x20c8908 (0x7f7d4ab5d908 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9957457Z frame #15: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f7d4a5580e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9958308Z frame #16: <unknown function> + 0x3443e11 (0x7f7d4bed8e11 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9958938Z frame #17: <unknown function> + 0x34443bb (0x7f7d4bed93bb in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9960057Z frame #18: at::_ops::_to_copy::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x201 (0x7f7d4a5b1de1 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9961196Z frame #19: at::native::to(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x13e (0x7f7d4a08c05e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9962037Z frame #20: <unknown function> + 0x2471e09 (0x7f7d4af06e09 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9962979Z frame #21: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x215 (0x7f7d4a7233f5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9963777Z frame #22: <unknown function> + 0x36467f (0x7f7d557d867f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:36.9964392Z frame #23: <unknown function> + 0x364b3c (0x7f7d557d8b3c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:36.9964865Z frame #24: <unknown function> + 0x1ddc68 (0x556a87974c68 in /opt/conda/bin/python)
2022-11-23T02:55:36.9965236Z frame #25: <unknown function> + 0x1049f3 (0x556a8789b9f3 in /opt/conda/bin/python)
2022-11-23T02:55:36.9965599Z frame #26: <unknown function> + 0x18f742 (0x556a87926742 in /opt/conda/bin/python)
2022-11-23T02:55:36.9965975Z frame #27: <unknown function> + 0x104425 (0x556a8789b425 in /opt/conda/bin/python)
2022-11-23T02:55:36.9966356Z frame #28: <unknown function> + 0x18f742 (0x556a87926742 in /opt/conda/bin/python)
2022-11-23T02:55:36.9966919Z frame #29: <unknown function> + 0x18fc9b (0x556a87926c9b in /opt/conda/bin/python)
2022-11-23T02:55:36.9967287Z frame #30: <unknown function> + 0x1052a5 (0x556a8789c2a5 in /opt/conda/bin/python)
2022-11-23T02:55:36.9967681Z frame #31: <unknown function> + 0x18f742 (0x556a87926742 in /opt/conda/bin/python)
2022-11-23T02:55:36.9968067Z frame #32: <unknown function> + 0x18fc9b (0x556a87926c9b in /opt/conda/bin/python)
2022-11-23T02:55:36.9968435Z frame #33: <unknown function> + 0x1052a5 (0x556a8789c2a5 in /opt/conda/bin/python)
2022-11-23T02:55:36.9968828Z frame #34: <unknown function> + 0x18f742 (0x556a87926742 in /opt/conda/bin/python)
2022-11-23T02:55:36.9969219Z frame #35: _PyObject_Call + 0x20a (0x556a878defaa in /opt/conda/bin/python)
2022-11-23T02:55:36.9969783Z frame #36: _PyEval_EvalFrameDefault + 0x26e4 (0x556a8797a774 in /opt/conda/bin/python)
2022-11-23T02:55:36.9970159Z frame #37: <unknown function> + 0x18f742 (0x556a87926742 in /opt/conda/bin/python)
2022-11-23T02:55:36.9970537Z frame #38: _PyObject_Call + 0x20a (0x556a878defaa in /opt/conda/bin/python)
2022-11-23T02:55:36.9971306Z frame #39: <unknown function> + 0xaa8dba (0x7f7d55f1cdba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:36.9972159Z frame #40: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f7d55f1affd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:36.9973170Z frame #41: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f7d55f1e2d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:36.9974441Z frame #42: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x83 (0x7f7d55f1e9a3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:36.9975821Z frame #43: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x194 (0x7f7d4d4c0654 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9977285Z frame #44: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f7d55f1e0c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:36.9978276Z frame #45: <unknown function> + 0x4a24a53 (0x7f7d4d4b9a53 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9979216Z frame #46: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f7d4d4ba5e8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9980271Z frame #47: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f7d4d4b48e7 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9981106Z frame #48: <unknown function> + 0x4a545d2 (0x7f7d4d4e95d2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:36.9981799Z frame #49: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f7d3d7b090b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:55:36.9982310Z frame #50: <unknown function> + 0xdbbf4 (0x7f7d6d616bf4 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-11-23T02:55:36.9982851Z frame #51: <unknown function> + 0x76db (0x7f7d8dc6a6db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-11-23T02:55:36.9983357Z frame #52: clone + 0x3f (0x7f7d8d99361f in /lib/x86_64-linux-gnu/libc.so.6)
2022-11-23T02:55:36.9983584Z 
2022-11-23T02:55:36.9983602Z 
2022-11-23T02:55:36.9983620Z 
2022-11-23T02:55:36.9983725Z ok (4.833s)
2022-11-23T02:55:36.9983874Z 
2022-11-23T02:55:36.9984150Z ----------------------------------------------------------------------
2022-11-23T02:55:36.9984462Z Ran 1 test in 4.834s
2022-11-23T02:55:36.9984629Z 
2022-11-23T02:55:36.9984725Z OK
2022-11-23T02:55:36.9984860Z 
2022-11-23T02:55:36.9984987Z Generating XML reports...
2022-11-23T02:55:36.9985668Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221123023405.xml
2022-11-23T02:55:36.9986569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:36.9987008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:36.9987568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:36.9988009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:36.9988535Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjhhudi4r
2022-11-23T02:55:36.9989252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjhhudi4r/_remote_module_non_scriptable.py
2022-11-23T02:55:36.9989560Z 
2022-11-23T02:55:36.9989671Z Running tests...
2022-11-23T02:55:36.9990067Z ----------------------------------------------------------------------
2022-11-23T02:55:36.9990644Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:36.9991196Z   test_valid_device (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:36.9991697Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75587
2022-11-23T02:55:36.9992279Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75588
2022-11-23T02:55:36.9992874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:36.9993317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:36.9993855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:36.9994363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:36.9995103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:36.9995549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:36.9996098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:36.9996559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:36.9997028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5l1w9aac
2022-11-23T02:55:36.9997546Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5l1w9aac/_remote_module_non_scriptable.py
2022-11-23T02:55:36.9998236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4omb56js
2022-11-23T02:55:36.9998763Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4omb56js/_remote_module_non_scriptable.py
2022-11-23T02:55:36.9999259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:36.9999700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0000081Z fi_getinfo: -61
2022-11-23T02:55:37.0000353Z fi_getinfo: -61
2022-11-23T02:55:37.0000565Z ok (6.213s)
2022-11-23T02:55:37.0000710Z 
2022-11-23T02:55:37.0000970Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0001289Z Ran 1 test in 6.214s
2022-11-23T02:55:37.0001447Z 
2022-11-23T02:55:37.0001520Z OK
2022-11-23T02:55:37.0001652Z 
2022-11-23T02:55:37.0001777Z Generating XML reports...
2022-11-23T02:55:37.0002433Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221123023412.xml
2022-11-23T02:55:37.0003183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0003602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0004160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0004622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0005078Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe5pvf2as
2022-11-23T02:55:37.0005587Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe5pvf2as/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0005881Z 
2022-11-23T02:55:37.0005988Z Running tests...
2022-11-23T02:55:37.0006383Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0006972Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0007503Z   test_profiler_remote_cuda (__main__.TensorPipeCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0007978Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75873
2022-11-23T02:55:37.0008418Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75874
2022-11-23T02:55:37.0008836Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 75875
2022-11-23T02:55:37.0009438Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 75876
2022-11-23T02:55:37.0010056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0010488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0011065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0011538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0012322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0012733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0013469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0013932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0014510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0014932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0015563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0016173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0017066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0017528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0018101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0018561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0019007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnlhldn_o
2022-11-23T02:55:37.0019553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnlhldn_o/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0020364Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpttqpr4m3
2022-11-23T02:55:37.0020763Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpttqpr4m3/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0021469Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps01qutj1
2022-11-23T02:55:37.0022009Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps01qutj1/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0022539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2j9ck6a8
2022-11-23T02:55:37.0023054Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2j9ck6a8/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0023561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0024037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0024508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0025037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0025434Z fi_getinfo: -61
2022-11-23T02:55:37.0025710Z fi_getinfo: -61
2022-11-23T02:55:37.0025969Z fi_getinfo: -61
2022-11-23T02:55:37.0026239Z fi_getinfo: -61
2022-11-23T02:55:37.0026487Z ok (9.115s)
2022-11-23T02:55:37.0026606Z 
2022-11-23T02:55:37.0026880Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0027209Z Ran 1 test in 9.115s
2022-11-23T02:55:37.0027372Z 
2022-11-23T02:55:37.0027464Z OK
2022-11-23T02:55:37.0027599Z 
2022-11-23T02:55:37.0027706Z Generating XML reports...
2022-11-23T02:55:37.0028359Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRpcTest-20221123023421.xml
2022-11-23T02:55:37.0029245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0029682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0030226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0030861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0031199Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi_gbvja5
2022-11-23T02:55:37.0031703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi_gbvja5/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0032161Z 
2022-11-23T02:55:37.0032279Z Running tests...
2022-11-23T02:55:37.0032680Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0033263Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0033790Z   test_basic_gloo_ckpt_always (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0034294Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76380
2022-11-23T02:55:37.0034755Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76381
2022-11-23T02:55:37.0035369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0035799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0036373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0036841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0037396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0037838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0038445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0039135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0039466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgy1jtsjn
2022-11-23T02:55:37.0040179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgy1jtsjn/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0040710Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwarzer7_
2022-11-23T02:55:37.0041288Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwarzer7_/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0041857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0042334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0042734Z fi_getinfo: -61
2022-11-23T02:55:37.0042989Z fi_getinfo: -61
2022-11-23T02:55:37.0043366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:55:37.0043940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:55:37.0044594Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:55:37.0045285Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:55:37.0045972Z [W logger.cpp:318] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-11-23T02:55:37.0046584Z [W logger.cpp:318] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-11-23T02:55:37.0047064Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:55:37.0047555Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:55:37.0047906Z ok (8.551s)
2022-11-23T02:55:37.0048061Z 
2022-11-23T02:55:37.0048435Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0048785Z Ran 1 test in 8.551s
2022-11-23T02:55:37.0049073Z 
2022-11-23T02:55:37.0049073Z OK
2022-11-23T02:55:37.0049180Z 
2022-11-23T02:55:37.0049304Z Generating XML reports...
2022-11-23T02:55:37.0049944Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123023433.xml
2022-11-23T02:55:37.0050693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0051146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0051724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0052177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0052652Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjx5ly1m7
2022-11-23T02:55:37.0053272Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjx5ly1m7/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0053577Z 
2022-11-23T02:55:37.0053666Z Running tests...
2022-11-23T02:55:37.0054071Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0054689Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0055205Z   test_basic_gloo_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0055842Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76687
2022-11-23T02:55:37.0056465Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76688
2022-11-23T02:55:37.0057331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0057767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0058343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0058817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0059397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0059820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0060388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0060848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0061361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj63cwrsg
2022-11-23T02:55:37.0062243Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj63cwrsg/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0062670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3v9ukv0h
2022-11-23T02:55:37.0063197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3v9ukv0h/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0063668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0064127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0064503Z fi_getinfo: -61
2022-11-23T02:55:37.0064882Z fi_getinfo: -61
2022-11-23T02:55:37.0065127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:55:37.0065761Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:55:37.0066283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:55:37.0067066Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:55:37.0067786Z [W logger.cpp:318] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-11-23T02:55:37.0068400Z [W logger.cpp:318] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-11-23T02:55:37.0068901Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:55:37.0069369Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:55:37.0069880Z ok (8.597s)
2022-11-23T02:55:37.0070026Z 
2022-11-23T02:55:37.0070289Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0070587Z Ran 1 test in 8.597s
2022-11-23T02:55:37.0070744Z 
2022-11-23T02:55:37.0070836Z OK
2022-11-23T02:55:37.0070966Z 
2022-11-23T02:55:37.0071090Z Generating XML reports...
2022-11-23T02:55:37.0071730Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123023445.xml
2022-11-23T02:55:37.0072447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0072886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0073444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0073891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0074349Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiboguw25
2022-11-23T02:55:37.0074879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiboguw25/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0075293Z 
2022-11-23T02:55:37.0075293Z Running tests...
2022-11-23T02:55:37.0075842Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0076424Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0076980Z   test_basic_gloo_ckpt_never (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0077482Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76994
2022-11-23T02:55:37.0077922Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76995
2022-11-23T02:55:37.0078528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0079135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0079853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0080380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0081006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0081457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0082017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0082480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0082947Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprpsvye5f
2022-11-23T02:55:37.0083469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprpsvye5f/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0084009Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5uhs9qrv
2022-11-23T02:55:37.0084545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5uhs9qrv/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0085062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0085576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0085969Z fi_getinfo: -61
2022-11-23T02:55:37.0086401Z fi_getinfo: -61
2022-11-23T02:55:37.0086755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:55:37.0087239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:55:37.0087878Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:55:37.0088548Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:55:37.0089327Z [W logger.cpp:318] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-11-23T02:55:37.0089947Z [W logger.cpp:318] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-11-23T02:55:37.0090448Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:55:37.0090935Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:55:37.0091270Z ok (8.610s)
2022-11-23T02:55:37.0091419Z 
2022-11-23T02:55:37.0091690Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0092021Z Ran 1 test in 8.610s
2022-11-23T02:55:37.0092336Z 
2022-11-23T02:55:37.0092513Z OK
2022-11-23T02:55:37.0092537Z 
2022-11-23T02:55:37.0092659Z Generating XML reports...
2022-11-23T02:55:37.0093293Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123023456.xml
2022-11-23T02:55:37.0094019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0094436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0094995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0095624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0096066Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsy_dedud
2022-11-23T02:55:37.0096609Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsy_dedud/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0097099Z 
2022-11-23T02:55:37.0097216Z Running tests...
2022-11-23T02:55:37.0097629Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0098184Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0098968Z   test_basic_gloo_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0099473Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77301
2022-11-23T02:55:37.0099900Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77302
2022-11-23T02:55:37.0100497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0100934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0101488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0101925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0102492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0102927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0103483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0103978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0104430Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnuc8ojk4
2022-11-23T02:55:37.0104956Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnuc8ojk4/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0105455Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp05akpd9_
2022-11-23T02:55:37.0105976Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp05akpd9_/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0106469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0106929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0107293Z fi_getinfo: -61
2022-11-23T02:55:37.0107565Z fi_getinfo: -61
2022-11-23T02:55:37.0107942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:55:37.0108569Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:55:37.0109090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:55:37.0109892Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:55:37.0110544Z [W logger.cpp:318] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-11-23T02:55:37.0111134Z [W logger.cpp:318] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-11-23T02:55:37.0111498Z ok (8.447s)
2022-11-23T02:55:37.0111647Z 
2022-11-23T02:55:37.0111921Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0112231Z Ran 1 test in 8.447s
2022-11-23T02:55:37.0112394Z 
2022-11-23T02:55:37.0112492Z OK
2022-11-23T02:55:37.0112780Z 
2022-11-23T02:55:37.0112901Z Generating XML reports...
2022-11-23T02:55:37.0113721Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123023508.xml
2022-11-23T02:55:37.0114454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0114911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0115489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0115937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0116549Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpny0jqxlx
2022-11-23T02:55:37.0117126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpny0jqxlx/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0117428Z 
2022-11-23T02:55:37.0117536Z Running tests...
2022-11-23T02:55:37.0117915Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0118469Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0118997Z   test_basic_nccl_ckpt_always (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0119479Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77608
2022-11-23T02:55:37.0119901Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77609
2022-11-23T02:55:37.0120481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0120918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0121640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0122169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0122752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0123201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0123750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0124213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0124821Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv074vtn9
2022-11-23T02:55:37.0125323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv074vtn9/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0125846Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmlcyjbcm
2022-11-23T02:55:37.0126369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmlcyjbcm/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0127046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0127498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0127887Z fi_getinfo: -61
2022-11-23T02:55:37.0128165Z fi_getinfo: -61
2022-11-23T02:55:37.0128529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:55:37.0129183Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:55:37.0129868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:55:37.0130498Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:55:37.0131106Z [W logger.cpp:318] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-11-23T02:55:37.0131704Z [W logger.cpp:318] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-11-23T02:55:37.0132185Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:55:37.0132835Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:55:37.0133170Z ok (9.822s)
2022-11-23T02:55:37.0133317Z 
2022-11-23T02:55:37.0133590Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0133919Z Ran 1 test in 9.822s
2022-11-23T02:55:37.0134080Z 
2022-11-23T02:55:37.0134153Z OK
2022-11-23T02:55:37.0134290Z 
2022-11-23T02:55:37.0134415Z Generating XML reports...
2022-11-23T02:55:37.0135121Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123023519.xml
2022-11-23T02:55:37.0135884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0136468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0137210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0137670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0138104Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu2v0trvr
2022-11-23T02:55:37.0138625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu2v0trvr/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0138919Z 
2022-11-23T02:55:37.0139026Z Running tests...
2022-11-23T02:55:37.0139421Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0140065Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0140582Z   test_basic_nccl_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0141251Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77926
2022-11-23T02:55:37.0141685Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77927
2022-11-23T02:55:37.0142298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0142750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0143328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0143784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0144517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0144954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0145508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0145968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0146424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt61gnia8
2022-11-23T02:55:37.0146945Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt61gnia8/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0147442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzdbgutat
2022-11-23T02:55:37.0148142Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzdbgutat/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0148656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0149130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0149511Z fi_getinfo: -61
2022-11-23T02:55:37.0149786Z fi_getinfo: -61
2022-11-23T02:55:37.0150170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:55:37.0150810Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:55:37.0151342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:55:37.0151994Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:55:37.0152640Z [W logger.cpp:318] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-11-23T02:55:37.0153297Z [W logger.cpp:318] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-11-23T02:55:37.0153809Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:55:37.0154316Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:55:37.0154656Z ok (9.802s)
2022-11-23T02:55:37.0154809Z 
2022-11-23T02:55:37.0155114Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0155425Z Ran 1 test in 9.802s
2022-11-23T02:55:37.0155742Z 
2022-11-23T02:55:37.0155834Z OK
2022-11-23T02:55:37.0155944Z 
2022-11-23T02:55:37.0156065Z Generating XML reports...
2022-11-23T02:55:37.0156882Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123023531.xml
2022-11-23T02:55:37.0157750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0158193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0158778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0159311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0159780Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfeqdp64a
2022-11-23T02:55:37.0160306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfeqdp64a/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0160767Z 
2022-11-23T02:55:37.0160884Z Running tests...
2022-11-23T02:55:37.0161339Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0161875Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0162409Z   test_basic_nccl_ckpt_never (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0162899Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78244
2022-11-23T02:55:37.0163366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78245
2022-11-23T02:55:37.0163924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0164358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0164948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0165375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0165916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0166344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0166904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0167516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0168004Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa6uxljpi
2022-11-23T02:55:37.0168560Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa6uxljpi/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0169107Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0lefksbe
2022-11-23T02:55:37.0169636Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0lefksbe/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0170305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0170777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0171142Z fi_getinfo: -61
2022-11-23T02:55:37.0171423Z fi_getinfo: -61
2022-11-23T02:55:37.0171868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:55:37.0172373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:55:37.0172998Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:55:37.0173682Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:55:37.0174314Z [W logger.cpp:318] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-11-23T02:55:37.0174917Z [W logger.cpp:318] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-11-23T02:55:37.0175384Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:55:37.0176049Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:55:37.0176159Z ok (9.700s)
2022-11-23T02:55:37.0176180Z 
2022-11-23T02:55:37.0176511Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0176632Z Ran 1 test in 9.700s
2022-11-23T02:55:37.0176652Z 
2022-11-23T02:55:37.0177114Z OK
2022-11-23T02:55:37.0177165Z 
2022-11-23T02:55:37.0177165Z Generating XML reports...
2022-11-23T02:55:37.0177660Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123023544.xml
2022-11-23T02:55:37.0178039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0178224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0178614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0178819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0179085Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ez9_tg4
2022-11-23T02:55:37.0179517Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ez9_tg4/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0179537Z 
2022-11-23T02:55:37.0179624Z Running tests...
2022-11-23T02:55:37.0180073Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0180431Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0180723Z   test_basic_nccl_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0180973Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78562
2022-11-23T02:55:37.0181218Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78563
2022-11-23T02:55:37.0181607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0181790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0182178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0182350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0182721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0182905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0183281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0183476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0183738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_qyvv1dj
2022-11-23T02:55:37.0184095Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_qyvv1dj/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0184368Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxnzyn0qh
2022-11-23T02:55:37.0184619Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxnzyn0qh/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0184855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0185090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0185244Z fi_getinfo: -61
2022-11-23T02:55:37.0185386Z fi_getinfo: -61
2022-11-23T02:55:37.0185638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:55:37.0185888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:55:37.0186299Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:55:37.0186896Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:55:37.0187350Z [W logger.cpp:318] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-11-23T02:55:37.0187609Z [W logger.cpp:318] Warning: Cuda time stats are not collected for multi-device modules. (function operator())
2022-11-23T02:55:37.0187698Z ok (9.799s)
2022-11-23T02:55:37.0187717Z 
2022-11-23T02:55:37.0187978Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0188093Z Ran 1 test in 9.799s
2022-11-23T02:55:37.0188111Z 
2022-11-23T02:55:37.0188205Z OK
2022-11-23T02:55:37.0188223Z 
2022-11-23T02:55:37.0188349Z Generating XML reports...
2022-11-23T02:55:37.0188844Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123023557.xml
2022-11-23T02:55:37.0189190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0189370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0189936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0190117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0190379Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoowt6i4z
2022-11-23T02:55:37.0190751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoowt6i4z/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0190751Z 
2022-11-23T02:55:37.0190798Z Running tests...
2022-11-23T02:55:37.0191070Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0191438Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0191746Z   test_async_execution_nested_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0191976Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78880
2022-11-23T02:55:37.0192201Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78881
2022-11-23T02:55:37.0192424Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 78882
2022-11-23T02:55:37.0192850Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 78883
2022-11-23T02:55:37.0193166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0193342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0193763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0193938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0194292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0194457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0194827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0195015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0195371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0195545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0196152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0196292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0196685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0196862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0197241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0197438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0197702Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpenphy86p
2022-11-23T02:55:37.0197988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpenphy86p/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0198250Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb6y83w2r
2022-11-23T02:55:37.0198526Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb6y83w2r/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0198913Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp160hxw7f
2022-11-23T02:55:37.0199181Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp160hxw7f/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0199431Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn7g4mjvc
2022-11-23T02:55:37.0199695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn7g4mjvc/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0199923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0200148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0200368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0200597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0200746Z fi_getinfo: -61
2022-11-23T02:55:37.0200863Z fi_getinfo: -61
2022-11-23T02:55:37.0201005Z fi_getinfo: -61
2022-11-23T02:55:37.0201142Z fi_getinfo: -61
2022-11-23T02:55:37.0201246Z ok (12.006s)
2022-11-23T02:55:37.0201265Z 
2022-11-23T02:55:37.0201525Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0201640Z Ran 1 test in 12.007s
2022-11-23T02:55:37.0201658Z 
2022-11-23T02:55:37.0201751Z OK
2022-11-23T02:55:37.0201770Z 
2022-11-23T02:55:37.0201871Z Generating XML reports...
2022-11-23T02:55:37.0202411Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023609.xml
2022-11-23T02:55:37.0202781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0202958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0203393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0203594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0203851Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwk7cyasu
2022-11-23T02:55:37.0204117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwk7cyasu/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0204136Z 
2022-11-23T02:55:37.0204247Z Running tests...
2022-11-23T02:55:37.0204491Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0204843Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0205157Z   test_async_execution_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0205378Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79387
2022-11-23T02:55:37.0205597Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79388
2022-11-23T02:55:37.0205864Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 79389
2022-11-23T02:55:37.0206081Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 79390
2022-11-23T02:55:37.0206448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0206599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0206970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0207166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0207525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0207703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0208069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0208261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0208615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0208796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0209136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0209326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0209682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0209859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0210413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0210733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0210874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjojsuhd0
2022-11-23T02:55:37.0211154Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjojsuhd0/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0211389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx9h8clh0
2022-11-23T02:55:37.0211665Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx9h8clh0/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0211930Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplmnx9oix
2022-11-23T02:55:37.0212204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplmnx9oix/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0212511Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv17bxz_o
2022-11-23T02:55:37.0212798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv17bxz_o/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0213190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0213414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0213808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0214020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0214174Z fi_getinfo: -61
2022-11-23T02:55:37.0214319Z fi_getinfo: -61
2022-11-23T02:55:37.0214464Z fi_getinfo: -61
2022-11-23T02:55:37.0214605Z fi_getinfo: -61
2022-11-23T02:55:37.0214709Z ok (12.007s)
2022-11-23T02:55:37.0214730Z 
2022-11-23T02:55:37.0215004Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0215099Z Ran 1 test in 12.007s
2022-11-23T02:55:37.0215166Z 
2022-11-23T02:55:37.0215271Z OK
2022-11-23T02:55:37.0215289Z 
2022-11-23T02:55:37.0215422Z Generating XML reports...
2022-11-23T02:55:37.0215980Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023624.xml
2022-11-23T02:55:37.0216361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0216869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0217259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0217454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0217715Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe6aigtat
2022-11-23T02:55:37.0217959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe6aigtat/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0217982Z 
2022-11-23T02:55:37.0218095Z Running tests...
2022-11-23T02:55:37.0218357Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0218709Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0219030Z   test_cuda_future_callback_changes_devices (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0219249Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79894
2022-11-23T02:55:37.0219467Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79895
2022-11-23T02:55:37.0219681Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 79896
2022-11-23T02:55:37.0219873Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 79897
2022-11-23T02:55:37.0220243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0220427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0220804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0220997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0221360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0221536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0222087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0222284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0222703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0223024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0223279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0223473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0223838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0224018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0224399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0224591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0224983Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ps8rqmz
2022-11-23T02:55:37.0225253Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ps8rqmz/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0225568Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9h383d4b
2022-11-23T02:55:37.0225833Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9h383d4b/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0226082Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp158hy3y4
2022-11-23T02:55:37.0226343Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp158hy3y4/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0226594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplenm_e5g
2022-11-23T02:55:37.0226854Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplenm_e5g/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0227263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0227468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0227708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0227939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0228050Z ok (11.437s)
2022-11-23T02:55:37.0228069Z 
2022-11-23T02:55:37.0228352Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0228473Z Ran 1 test in 11.437s
2022-11-23T02:55:37.0228492Z 
2022-11-23T02:55:37.0228592Z OK
2022-11-23T02:55:37.0228611Z 
2022-11-23T02:55:37.0228741Z Generating XML reports...
2022-11-23T02:55:37.0229269Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023639.xml
2022-11-23T02:55:37.0229654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0229996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0230369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0230561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0230812Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo4lmitsi
2022-11-23T02:55:37.0231077Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo4lmitsi/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0231096Z 
2022-11-23T02:55:37.0231211Z Running tests...
2022-11-23T02:55:37.0231473Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0231801Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0232214Z   test_cuda_future_can_extract_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0232408Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80233
2022-11-23T02:55:37.0232629Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80234
2022-11-23T02:55:37.0233016Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 80235
2022-11-23T02:55:37.0233244Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 80236
2022-11-23T02:55:37.0233627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0233812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0234178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0234382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0234758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0234988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0235373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0235569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0235940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0236121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0236502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0236670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0237041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0237227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0237612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0237807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0238073Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi341sppn
2022-11-23T02:55:37.0238349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi341sppn/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0238611Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp07oqzm27
2022-11-23T02:55:37.0238859Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp07oqzm27/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0239223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfrakhp7w
2022-11-23T02:55:37.0239401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfrakhp7w/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0239662Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwk26jcgz
2022-11-23T02:55:37.0239932Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwk26jcgz/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0240166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0240396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0240633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0241013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0241093Z ok (10.150s)
2022-11-23T02:55:37.0241113Z 
2022-11-23T02:55:37.0241613Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0241743Z Ran 1 test in 10.150s
2022-11-23T02:55:37.0241763Z 
2022-11-23T02:55:37.0241862Z OK
2022-11-23T02:55:37.0241881Z 
2022-11-23T02:55:37.0242009Z Generating XML reports...
2022-11-23T02:55:37.0242566Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023653.xml
2022-11-23T02:55:37.0242942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0243124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0243510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0243684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0243952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp45nyhjz
2022-11-23T02:55:37.0244230Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp45nyhjz/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0244451Z 
2022-11-23T02:55:37.0244565Z Running tests...
2022-11-23T02:55:37.0244833Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0245186Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0245501Z   test_cuda_future_can_extract_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0245813Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80692
2022-11-23T02:55:37.0246094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80693
2022-11-23T02:55:37.0246317Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 80694
2022-11-23T02:55:37.0246541Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 80695
2022-11-23T02:55:37.0246919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0247104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0247482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0247664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0248049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0248249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0248608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0248811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0249182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0249367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0249814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0249940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0250312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0250497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0250855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0251048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0251358Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpky1hsb9e
2022-11-23T02:55:37.0251643Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpky1hsb9e/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0251907Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcdz8doc8
2022-11-23T02:55:37.0252179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcdz8doc8/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0252445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptz2hw88i
2022-11-23T02:55:37.0252708Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsbkslgoj
2022-11-23T02:55:37.0252981Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptz2hw88i/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0253227Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsbkslgoj/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0253467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0253699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0253980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0254210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0254319Z ok (10.146s)
2022-11-23T02:55:37.0254339Z 
2022-11-23T02:55:37.0254619Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0254739Z Ran 1 test in 10.147s
2022-11-23T02:55:37.0254758Z 
2022-11-23T02:55:37.0254831Z OK
2022-11-23T02:55:37.0254875Z 
2022-11-23T02:55:37.0254980Z Generating XML reports...
2022-11-23T02:55:37.0255535Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023706.xml
2022-11-23T02:55:37.0256074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0256254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0256960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0257167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0257430Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxcg6cu3j
2022-11-23T02:55:37.0257710Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxcg6cu3j/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0257731Z 
2022-11-23T02:55:37.0257820Z Running tests...
2022-11-23T02:55:37.0258100Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0258468Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0258829Z   test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0259056Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81027
2022-11-23T02:55:37.0259282Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81028
2022-11-23T02:55:37.0259504Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 81029
2022-11-23T02:55:37.0259725Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 81030
2022-11-23T02:55:37.0260107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0260265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0260650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0261241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0261626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0261807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0262175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0262364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0262722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0262924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0263241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0263433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0263789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0264063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0264439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0264627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0265000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpur_07i12
2022-11-23T02:55:37.0265148Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpur_07i12/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0265376Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp52fn6ydt
2022-11-23T02:55:37.0265645Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp52fn6ydt/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0265904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplq00nj6j
2022-11-23T02:55:37.0266170Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplq00nj6j/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0266422Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsnwj0pp6
2022-11-23T02:55:37.0266689Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsnwj0pp6/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0266915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0267134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0267335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0267565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0267842Z ok (10.103s)
2022-11-23T02:55:37.0267865Z 
2022-11-23T02:55:37.0268145Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0268270Z Ran 1 test in 10.104s
2022-11-23T02:55:37.0268290Z 
2022-11-23T02:55:37.0268387Z OK
2022-11-23T02:55:37.0268406Z 
2022-11-23T02:55:37.0268535Z Generating XML reports...
2022-11-23T02:55:37.0269089Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023719.xml
2022-11-23T02:55:37.0269466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0269623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0270011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0270207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0270693Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgjpccfcq
2022-11-23T02:55:37.0271144Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgjpccfcq/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0271164Z 
2022-11-23T02:55:37.0271279Z Running tests...
2022-11-23T02:55:37.0271552Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0271918Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0272240Z   test_cuda_future_can_extract_custom_class_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0272469Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81490
2022-11-23T02:55:37.0272699Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81491
2022-11-23T02:55:37.0273047Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 81492
2022-11-23T02:55:37.0273150Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 81493
2022-11-23T02:55:37.0273580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0273761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0274294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0274487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0274825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0275002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0275367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0275557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0275918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0276095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0276641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0276835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0277175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0277356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0277831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0277935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0278204Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpampp3kta
2022-11-23T02:55:37.0278484Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpampp3kta/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0278744Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgrebpnnv
2022-11-23T02:55:37.0279022Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgrebpnnv/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0279283Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa1yp_jmv
2022-11-23T02:55:37.0279531Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa1yp_jmv/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0279940Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy_rgn2s2
2022-11-23T02:55:37.0280429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy_rgn2s2/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0280670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0280905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0281188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0281423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0281532Z ok (10.103s)
2022-11-23T02:55:37.0281552Z 
2022-11-23T02:55:37.0281830Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0281924Z Ran 1 test in 10.103s
2022-11-23T02:55:37.0281942Z 
2022-11-23T02:55:37.0282042Z OK
2022-11-23T02:55:37.0282060Z 
2022-11-23T02:55:37.0282190Z Generating XML reports...
2022-11-23T02:55:37.0282752Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023732.xml
2022-11-23T02:55:37.0283129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0283365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0283750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0283949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0284189Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjow7jhw0
2022-11-23T02:55:37.0284467Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjow7jhw0/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0284487Z 
2022-11-23T02:55:37.0284604Z Running tests...
2022-11-23T02:55:37.0284874Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0285242Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0285596Z   test_cuda_future_can_extract_list_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0285827Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81829
2022-11-23T02:55:37.0286056Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81830
2022-11-23T02:55:37.0286280Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 81831
2022-11-23T02:55:37.0286476Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 81832
2022-11-23T02:55:37.0287009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0287187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0287565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0287760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0288129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0288306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0288670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0288834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0289191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0289367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0289724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0290120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0290512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0290715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0291132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0291296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0291538Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfpsrnk5j
2022-11-23T02:55:37.0291816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfpsrnk5j/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0292078Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt_738r3u
2022-11-23T02:55:37.0292353Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt_738r3u/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0292613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpirj315m9
2022-11-23T02:55:37.0293088Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpirj315m9/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0293341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq2og8l4s
2022-11-23T02:55:37.0293605Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq2og8l4s/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0293809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0294032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0294259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0294487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0294591Z ok (10.103s)
2022-11-23T02:55:37.0294612Z 
2022-11-23T02:55:37.0294887Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0295008Z Ran 1 test in 10.103s
2022-11-23T02:55:37.0295027Z 
2022-11-23T02:55:37.0295121Z OK
2022-11-23T02:55:37.0295139Z 
2022-11-23T02:55:37.0295265Z Generating XML reports...
2022-11-23T02:55:37.0295776Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023745.xml
2022-11-23T02:55:37.0296315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0296525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0297066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0297271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0297533Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgac4bmoz
2022-11-23T02:55:37.0297813Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgac4bmoz/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0297833Z 
2022-11-23T02:55:37.0297951Z Running tests...
2022-11-23T02:55:37.0298227Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0298567Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0298906Z   test_cuda_future_can_extract_list_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0299284Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82288
2022-11-23T02:55:37.0299503Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82289
2022-11-23T02:55:37.0299879Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 82290
2022-11-23T02:55:37.0300015Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 82291
2022-11-23T02:55:37.0300389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0300567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0300916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0301107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0301465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0301638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0302009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0302197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0302616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0302793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0303159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0303321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0303679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0303851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0304221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0304413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0304669Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph22g_uhq
2022-11-23T02:55:37.0304937Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph22g_uhq/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0305193Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe8ygtnp3
2022-11-23T02:55:37.0305435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe8ygtnp3/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0305688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdag7lz9z
2022-11-23T02:55:37.0305952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdag7lz9z/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0306202Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp806aif9k
2022-11-23T02:55:37.0306468Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp806aif9k/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0306700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0306925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0307150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0307379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0307460Z ok (10.005s)
2022-11-23T02:55:37.0307478Z 
2022-11-23T02:55:37.0307750Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0307865Z Ran 1 test in 10.005s
2022-11-23T02:55:37.0307884Z 
2022-11-23T02:55:37.0307982Z OK
2022-11-23T02:55:37.0308000Z 
2022-11-23T02:55:37.0308125Z Generating XML reports...
2022-11-23T02:55:37.0308724Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023758.xml
2022-11-23T02:55:37.0309098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0309278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0309626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0309823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0310081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmwmnv8hb
2022-11-23T02:55:37.0310349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmwmnv8hb/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0310535Z 
2022-11-23T02:55:37.0310666Z Running tests...
2022-11-23T02:55:37.0310918Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0311287Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0311659Z   test_cuda_future_device_as_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0311883Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82623
2022-11-23T02:55:37.0312083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82624
2022-11-23T02:55:37.0312310Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 82625
2022-11-23T02:55:37.0312534Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 82626
2022-11-23T02:55:37.0312972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0313103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0313641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0313837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0314380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0314538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0314917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0315115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0315486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0315666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0316044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0316239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0316616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0316794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0317296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0317482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0317768Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmwd7ixdf
2022-11-23T02:55:37.0318012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmwd7ixdf/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0318258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9cvbbf94
2022-11-23T02:55:37.0318569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9cvbbf94/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0318838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1govxuru
2022-11-23T02:55:37.0319102Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1govxuru/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0319326Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv3lq5_0o
2022-11-23T02:55:37.0319585Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv3lq5_0o/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0319809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0320030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0320251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0320475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0320576Z ok (4.548s)
2022-11-23T02:55:37.0320597Z 
2022-11-23T02:55:37.0320914Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0321027Z Ran 1 test in 4.548s
2022-11-23T02:55:37.0321046Z 
2022-11-23T02:55:37.0321116Z OK
2022-11-23T02:55:37.0321134Z 
2022-11-23T02:55:37.0321262Z Generating XML reports...
2022-11-23T02:55:37.0321794Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023811.xml
2022-11-23T02:55:37.0322338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0322516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0322893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0323093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0323364Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp42cwka6o
2022-11-23T02:55:37.0323603Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp42cwka6o/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0323646Z 
2022-11-23T02:55:37.0323736Z Running tests...
2022-11-23T02:55:37.0324001Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0324363Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0324673Z   test_cuda_future_device_as_int (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0324896Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82954
2022-11-23T02:55:37.0325117Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82955
2022-11-23T02:55:37.0325336Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 82956
2022-11-23T02:55:37.0325554Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 82957
2022-11-23T02:55:37.0325915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0326139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0326523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0326716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0327086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0327261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0327685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0327881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0328232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0328409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0328782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0328973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0329340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0329508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0329887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0330081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0330537Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1869o733
2022-11-23T02:55:37.0330770Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf2cal5ch
2022-11-23T02:55:37.0331035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1869o733/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0331291Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf2cal5ch/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0331544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnjs8dmbl
2022-11-23T02:55:37.0331925Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnjs8dmbl/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0332046Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsd_sxsyy
2022-11-23T02:55:37.0332307Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsd_sxsyy/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0332571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0332749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0333121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0333357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0333456Z ok (4.477s)
2022-11-23T02:55:37.0333476Z 
2022-11-23T02:55:37.0333751Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0333866Z Ran 1 test in 4.477s
2022-11-23T02:55:37.0333885Z 
2022-11-23T02:55:37.0333978Z OK
2022-11-23T02:55:37.0333997Z 
2022-11-23T02:55:37.0334125Z Generating XML reports...
2022-11-23T02:55:37.0334678Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023818.xml
2022-11-23T02:55:37.0335032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0335217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0335597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0335791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0336044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6pc4mgel
2022-11-23T02:55:37.0336317Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6pc4mgel/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0336337Z 
2022-11-23T02:55:37.0336448Z Running tests...
2022-11-23T02:55:37.0336886Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0337335Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0337637Z   test_cuda_future_device_as_str (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0337860Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83285
2022-11-23T02:55:37.0338080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83286
2022-11-23T02:55:37.0338299Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 83287
2022-11-23T02:55:37.0338513Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 83288
2022-11-23T02:55:37.0338892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0339226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0339617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0339769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0340191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0340540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0340916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0341106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0341469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0341729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0342139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0342328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0342672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0342849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0343228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0343419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0343678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp58x3ksk3
2022-11-23T02:55:37.0343947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp58x3ksk3/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0344204Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpurm5h_75
2022-11-23T02:55:37.0344477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpurm5h_75/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0344714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo066uu96
2022-11-23T02:55:37.0344983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo066uu96/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0345235Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5dw7jaer
2022-11-23T02:55:37.0345500Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5dw7jaer/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0345731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0345958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0346188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0346468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0346676Z ok (4.482s)
2022-11-23T02:55:37.0346676Z 
2022-11-23T02:55:37.0347168Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0347285Z Ran 1 test in 4.482s
2022-11-23T02:55:37.0347304Z 
2022-11-23T02:55:37.0347402Z OK
2022-11-23T02:55:37.0347418Z 
2022-11-23T02:55:37.0347548Z Generating XML reports...
2022-11-23T02:55:37.0348101Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023825.xml
2022-11-23T02:55:37.0348474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0348651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0349029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0349207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0349466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4gd5zr7s
2022-11-23T02:55:37.0349791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4gd5zr7s/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0349811Z 
2022-11-23T02:55:37.0349926Z Running tests...
2022-11-23T02:55:37.0350195Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0350639Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0350973Z   test_cuda_future_device_not_cuda (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0351097Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83616
2022-11-23T02:55:37.0351323Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83617
2022-11-23T02:55:37.0351526Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 83618
2022-11-23T02:55:37.0351745Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 83619
2022-11-23T02:55:37.0352124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0352303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0352684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0352877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0353244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0353543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0353782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0353972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0354342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0354517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0354886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0355078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0355440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0355616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0355998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0356216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0356635Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp070mfyzu
2022-11-23T02:55:37.0356900Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp070mfyzu/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0357320Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphuxe9mov
2022-11-23T02:55:37.0357594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphuxe9mov/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0357854Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptuvklusx
2022-11-23T02:55:37.0358128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptuvklusx/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0358382Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7b7jgwn8
2022-11-23T02:55:37.0358635Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7b7jgwn8/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0358867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0359159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0359389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0359618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0359723Z ok (4.507s)
2022-11-23T02:55:37.0359753Z 
2022-11-23T02:55:37.0360020Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0360139Z Ran 1 test in 4.507s
2022-11-23T02:55:37.0360159Z 
2022-11-23T02:55:37.0360254Z OK
2022-11-23T02:55:37.0360273Z 
2022-11-23T02:55:37.0360378Z Generating XML reports...
2022-11-23T02:55:37.0360933Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023833.xml
2022-11-23T02:55:37.0361364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0361551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0361932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0362128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0362390Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpouvp1rn_
2022-11-23T02:55:37.0362809Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpouvp1rn_/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0362955Z 
2022-11-23T02:55:37.0362969Z Running tests...
2022-11-23T02:55:37.0363229Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0363532Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0363844Z   test_cuda_future_modify_tensor_inplace (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0364064Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83947
2022-11-23T02:55:37.0364276Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83948
2022-11-23T02:55:37.0364489Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 83949
2022-11-23T02:55:37.0364698Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 83950
2022-11-23T02:55:37.0365064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0365216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0365633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0365827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0366193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0366364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0366726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0366911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0367261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0367434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0367777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0367960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0368546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0368722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0369209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0369287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0369545Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3j0mdkcl
2022-11-23T02:55:37.0369815Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3j0mdkcl/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0370157Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphu444sg8
2022-11-23T02:55:37.0370320Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphu444sg8/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0370581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7jq3253i
2022-11-23T02:55:37.0370839Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphgdc6kau
2022-11-23T02:55:37.0371257Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7jq3253i/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0371517Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphgdc6kau/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0371741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0371959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0372179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0372384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0372488Z ok (6.327s)
2022-11-23T02:55:37.0372509Z 
2022-11-23T02:55:37.0372771Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0372880Z Ran 1 test in 6.327s
2022-11-23T02:55:37.0372899Z 
2022-11-23T02:55:37.0372991Z OK
2022-11-23T02:55:37.0373010Z 
2022-11-23T02:55:37.0373131Z Generating XML reports...
2022-11-23T02:55:37.0373661Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023840.xml
2022-11-23T02:55:37.0374024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0374175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0374540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0374787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0375043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsbe3zbfm
2022-11-23T02:55:37.0375310Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsbe3zbfm/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0375329Z 
2022-11-23T02:55:37.0375439Z Running tests...
2022-11-23T02:55:37.0375769Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0376046Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0376347Z   test_cuda_future_replace_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0376541Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84282
2022-11-23T02:55:37.0377110Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84283
2022-11-23T02:55:37.0377338Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 84284
2022-11-23T02:55:37.0377561Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 84285
2022-11-23T02:55:37.0378016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0378191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0378573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0378767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0379116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0379292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0379665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0379856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0380374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0380711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0381121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0381314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0381672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0381826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0382203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0382393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0382653Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfxm8nnwh
2022-11-23T02:55:37.0382922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfxm8nnwh/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0383178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqwql7dv_
2022-11-23T02:55:37.0383447Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqwql7dv_/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0383703Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmped6g5kr3
2022-11-23T02:55:37.0383950Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmped6g5kr3/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0384204Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfcj0z_tk
2022-11-23T02:55:37.0384533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfcj0z_tk/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0384775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0384999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0385226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0385454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0385554Z ok (6.178s)
2022-11-23T02:55:37.0385573Z 
2022-11-23T02:55:37.0385847Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0385942Z Ran 1 test in 6.178s
2022-11-23T02:55:37.0385961Z 
2022-11-23T02:55:37.0386054Z OK
2022-11-23T02:55:37.0386072Z 
2022-11-23T02:55:37.0386201Z Generating XML reports...
2022-11-23T02:55:37.0386751Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023849.xml
2022-11-23T02:55:37.0387199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0387528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0387908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0388084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0388315Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsl1hdz1k
2022-11-23T02:55:37.0388582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsl1hdz1k/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0388601Z 
2022-11-23T02:55:37.0388709Z Running tests...
2022-11-23T02:55:37.0388962Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0389309Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0389618Z   test_cuda_future_value_on_bad_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0389824Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84617
2022-11-23T02:55:37.0390035Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84618
2022-11-23T02:55:37.0390240Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 84619
2022-11-23T02:55:37.0390597Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 84620
2022-11-23T02:55:37.0390969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0391139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0391521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0391710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0392073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0392240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0392605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0392868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0393133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0393478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0393885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0394067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0394426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0394599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0394958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0395140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0395369Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt4qkgxxa
2022-11-23T02:55:37.0395632Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt4qkgxxa/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0395870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpweccbpuy
2022-11-23T02:55:37.0396135Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpweccbpuy/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0396594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmgjgww5g
2022-11-23T02:55:37.0396903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmgjgww5g/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0397119Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz37h622z
2022-11-23T02:55:37.0397386Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz37h622z/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0397612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0397819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0398039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0398268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0398362Z ok (11.549s)
2022-11-23T02:55:37.0398385Z 
2022-11-23T02:55:37.0398655Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0398771Z Ran 1 test in 11.549s
2022-11-23T02:55:37.0398790Z 
2022-11-23T02:55:37.0398879Z OK
2022-11-23T02:55:37.0398898Z 
2022-11-23T02:55:37.0399021Z Generating XML reports...
2022-11-23T02:55:37.0399759Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023858.xml
2022-11-23T02:55:37.0400109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0400271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0400633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0400822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0401071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyistc8hg
2022-11-23T02:55:37.0401335Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyistc8hg/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0401353Z 
2022-11-23T02:55:37.0401455Z Running tests...
2022-11-23T02:55:37.0401698Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0402029Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0402305Z   test_custom_stream (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0403134Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/79750 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.714s)
2022-11-23T02:55:37.0403134Z 
2022-11-23T02:55:37.0403371Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0403477Z Ran 1 test in 1.714s
2022-11-23T02:55:37.0403496Z 
2022-11-23T02:55:37.0403599Z OK (skipped=1)
2022-11-23T02:55:37.0403617Z 
2022-11-23T02:55:37.0403733Z Generating XML reports...
2022-11-23T02:55:37.0404256Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023912.xml
2022-11-23T02:55:37.0404612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0404785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0405133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0405318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0405566Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu4p1m89i
2022-11-23T02:55:37.0405985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu4p1m89i/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0406004Z 
2022-11-23T02:55:37.0406107Z Running tests...
2022-11-23T02:55:37.0406359Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0406705Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0406986Z   test_custom_stream_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0407178Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85022
2022-11-23T02:55:37.0407387Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85023
2022-11-23T02:55:37.0407589Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 85024
2022-11-23T02:55:37.0407789Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 85025
2022-11-23T02:55:37.0408146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0408310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0408673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0408846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0409197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0409347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0409705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0409883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0410235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0410395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0410911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0411096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0411546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0411612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0411980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0412252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0412510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp29lb7bbe
2022-11-23T02:55:37.0412780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp29lb7bbe/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0413033Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ucaty2o
2022-11-23T02:55:37.0413299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ucaty2o/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0413549Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxm3_4zba
2022-11-23T02:55:37.0413965Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxm3_4zba/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0414191Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqwxdm46b
2022-11-23T02:55:37.0414612Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqwxdm46b/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0414833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0415102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0415331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0415554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0415706Z fi_getinfo: -61
2022-11-23T02:55:37.0415847Z fi_getinfo: -61
2022-11-23T02:55:37.0415964Z fi_getinfo: -61
2022-11-23T02:55:37.0416101Z fi_getinfo: -61
2022-11-23T02:55:37.0416197Z ok (19.754s)
2022-11-23T02:55:37.0416216Z 
2022-11-23T02:55:37.0416474Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0416578Z Ran 1 test in 19.754s
2022-11-23T02:55:37.0416596Z 
2022-11-23T02:55:37.0416798Z OK
2022-11-23T02:55:37.0417162Z 
2022-11-23T02:55:37.0417217Z Generating XML reports...
2022-11-23T02:55:37.0417846Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023917.xml
2022-11-23T02:55:37.0418196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0418369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0418822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0418908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0419148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6kbpapg6
2022-11-23T02:55:37.0419404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6kbpapg6/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0419426Z 
2022-11-23T02:55:37.0419523Z Running tests...
2022-11-23T02:55:37.0419768Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0420101Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0420392Z   test_custom_stream_nested (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0420597Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85541
2022-11-23T02:55:37.0420805Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85542
2022-11-23T02:55:37.0421008Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 85543
2022-11-23T02:55:37.0421204Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 85544
2022-11-23T02:55:37.0421553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0421788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0422167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0422506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0422872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0423047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0423412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0423593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0423953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0424122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0424483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0424714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0425078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0425253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0425776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0425959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0426208Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpavca7hfx
2022-11-23T02:55:37.0426469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpavca7hfx/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0426714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_xx3ljoz
2022-11-23T02:55:37.0426969Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_xx3ljoz/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0427197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzb4dbeov
2022-11-23T02:55:37.0427456Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzb4dbeov/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0427869Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmvuwen9y
2022-11-23T02:55:37.0428134Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmvuwen9y/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0428364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0428579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0428809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0429039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0429168Z fi_getinfo: -61
2022-11-23T02:55:37.0429297Z fi_getinfo: -61
2022-11-23T02:55:37.0429431Z fi_getinfo: -61
2022-11-23T02:55:37.0429557Z fi_getinfo: -61
2022-11-23T02:55:37.0429652Z ok (13.411s)
2022-11-23T02:55:37.0429671Z 
2022-11-23T02:55:37.0429933Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0430037Z Ran 1 test in 13.411s
2022-11-23T02:55:37.0430055Z 
2022-11-23T02:55:37.0430127Z OK
2022-11-23T02:55:37.0430166Z 
2022-11-23T02:55:37.0430273Z Generating XML reports...
2022-11-23T02:55:37.0430976Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023940.xml
2022-11-23T02:55:37.0431386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0431564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0431930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0432113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0432360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz7ta7ssv
2022-11-23T02:55:37.0432613Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz7ta7ssv/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0432633Z 
2022-11-23T02:55:37.0432720Z Running tests...
2022-11-23T02:55:37.0432974Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0433312Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0433785Z   test_custom_stream_nested_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0434054Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86060
2022-11-23T02:55:37.0434264Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86061
2022-11-23T02:55:37.0434481Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 86062
2022-11-23T02:55:37.0434687Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 86063
2022-11-23T02:55:37.0435059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0435216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0435596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0435783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0436158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0436334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0436701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0436891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0437396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0437547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0437908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0438088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0438436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0438606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0438969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0439138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0439389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf52jmh_a
2022-11-23T02:55:37.0439652Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf52jmh_a/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0440002Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv783r4ms
2022-11-23T02:55:37.0440133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv783r4ms/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0440444Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp_icyctt
2022-11-23T02:55:37.0440702Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp_icyctt/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0440950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2fjxionm
2022-11-23T02:55:37.0441206Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2fjxionm/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0441420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0441635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0441836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0442054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0442358Z fi_getinfo: -61
2022-11-23T02:55:37.0442497Z fi_getinfo: -61
2022-11-23T02:55:37.0442630Z fi_getinfo: -61
2022-11-23T02:55:37.0442758Z fi_getinfo: -61
2022-11-23T02:55:37.0442858Z ok (11.902s)
2022-11-23T02:55:37.0442922Z 
2022-11-23T02:55:37.0443178Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0443289Z Ran 1 test in 11.902s
2022-11-23T02:55:37.0443307Z 
2022-11-23T02:55:37.0443404Z OK
2022-11-23T02:55:37.0443423Z 
2022-11-23T02:55:37.0443547Z Generating XML reports...
2022-11-23T02:55:37.0444087Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023956.xml
2022-11-23T02:55:37.0444457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0444626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0445168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0445353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0445581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_17laugo
2022-11-23T02:55:37.0445832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_17laugo/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0445851Z 
2022-11-23T02:55:37.0445960Z Running tests...
2022-11-23T02:55:37.0446214Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0446556Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0446839Z   test_device_map_cpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0447047Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86574
2022-11-23T02:55:37.0447256Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86575
2022-11-23T02:55:37.0447469Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 86576
2022-11-23T02:55:37.0447663Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 86577
2022-11-23T02:55:37.0448024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0448188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0448551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0448727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0449262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0449433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0449854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0450033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0450406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0450580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0450944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0451126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0451476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0451763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0452105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0452211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0452504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp64h1l5bs
2022-11-23T02:55:37.0453086Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp64h1l5bs/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0453342Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph58yhzcb
2022-11-23T02:55:37.0453607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph58yhzcb/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0453861Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm5sg45rj
2022-11-23T02:55:37.0454124Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm5sg45rj/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0454366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk9i7dhm4
2022-11-23T02:55:37.0454632Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk9i7dhm4/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0454847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0455064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0455281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0455501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0455643Z fi_getinfo: -61
2022-11-23T02:55:37.0455777Z fi_getinfo: -61
2022-11-23T02:55:37.0455909Z fi_getinfo: -61
2022-11-23T02:55:37.0456036Z fi_getinfo: -61
2022-11-23T02:55:37.0456252Z ok (5.030s)
2022-11-23T02:55:37.0456270Z 
2022-11-23T02:55:37.0456402Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0456509Z Ran 1 test in 5.030s
2022-11-23T02:55:37.0456529Z 
2022-11-23T02:55:37.0456931Z OK
2022-11-23T02:55:37.0456954Z 
2022-11-23T02:55:37.0457078Z Generating XML reports...
2022-11-23T02:55:37.0457792Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024010.xml
2022-11-23T02:55:37.0458159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0458334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0458694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0458882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0459135Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph7d9lyfb
2022-11-23T02:55:37.0459400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph7d9lyfb/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0459490Z 
2022-11-23T02:55:37.0459596Z Running tests...
2022-11-23T02:55:37.0459859Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0460214Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0460527Z   test_device_map_cpu_to_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0460745Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87077
2022-11-23T02:55:37.0460944Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87078
2022-11-23T02:55:37.0461151Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 87079
2022-11-23T02:55:37.0461416Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 87080
2022-11-23T02:55:37.0462015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0462219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0462619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0462802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0463147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0463298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0463682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0463829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0464174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0464410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0464692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0464873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0465220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0465389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0465728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0465955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0466145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu0khogcz
2022-11-23T02:55:37.0466411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu0khogcz/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0466647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3q3uxc_t
2022-11-23T02:55:37.0466909Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3q3uxc_t/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0467153Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqefi2r59
2022-11-23T02:55:37.0467405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqefi2r59/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0467632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp34vo80_z
2022-11-23T02:55:37.0467885Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp34vo80_z/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0468101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0468311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0468750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0468980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0469123Z fi_getinfo: -61
2022-11-23T02:55:37.0469256Z fi_getinfo: -61
2022-11-23T02:55:37.0469373Z fi_getinfo: -61
2022-11-23T02:55:37.0469502Z fi_getinfo: -61
2022-11-23T02:55:37.0469598Z ok (8.016s)
2022-11-23T02:55:37.0469618Z 
2022-11-23T02:55:37.0469873Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0469984Z Ran 1 test in 8.016s
2022-11-23T02:55:37.0470003Z 
2022-11-23T02:55:37.0470095Z OK
2022-11-23T02:55:37.0470114Z 
2022-11-23T02:55:37.0470232Z Generating XML reports...
2022-11-23T02:55:37.0470780Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024018.xml
2022-11-23T02:55:37.0471140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0471516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0471884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0472065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0472309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk9plr4oa
2022-11-23T02:55:37.0472569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk9plr4oa/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0472589Z 
2022-11-23T02:55:37.0472690Z Running tests...
2022-11-23T02:55:37.0472938Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0473265Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0473581Z   test_device_map_cpu_to_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0473802Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87588
2022-11-23T02:55:37.0473997Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87589
2022-11-23T02:55:37.0474203Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 87590
2022-11-23T02:55:37.0474401Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 87591
2022-11-23T02:55:37.0474759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0474926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0475278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0475434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0475804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0475986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0476342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0476518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0476863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0477196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0477562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0477746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0478140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0478326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0478705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0478886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0479141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0sf15luw
2022-11-23T02:55:37.0479398Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnb55s5sl
2022-11-23T02:55:37.0479666Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0sf15luw/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0479928Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnb55s5sl/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0480168Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu8rbbml5
2022-11-23T02:55:37.0480645Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu8rbbml5/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0480891Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk974jmpq
2022-11-23T02:55:37.0481364Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk974jmpq/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0481598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0481818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0482052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0482272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0482424Z fi_getinfo: -61
2022-11-23T02:55:37.0482545Z fi_getinfo: -61
2022-11-23T02:55:37.0482687Z fi_getinfo: -61
2022-11-23T02:55:37.0482818Z fi_getinfo: -61
2022-11-23T02:55:37.0482924Z ok (8.023s)
2022-11-23T02:55:37.0482943Z 
2022-11-23T02:55:37.0483208Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0483309Z Ran 1 test in 8.023s
2022-11-23T02:55:37.0483329Z 
2022-11-23T02:55:37.0483419Z OK
2022-11-23T02:55:37.0483438Z 
2022-11-23T02:55:37.0483543Z Generating XML reports...
2022-11-23T02:55:37.0484091Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024029.xml
2022-11-23T02:55:37.0484458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0484633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0485007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0485199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0485455Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3li24j94
2022-11-23T02:55:37.0485714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3li24j94/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0485734Z 
2022-11-23T02:55:37.0485843Z Running tests...
2022-11-23T02:55:37.0486089Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0486444Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0486744Z   test_device_map_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0486964Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88099
2022-11-23T02:55:37.0487227Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88100
2022-11-23T02:55:37.0487452Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 88101
2022-11-23T02:55:37.0487824Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 88102
2022-11-23T02:55:37.0488185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0488443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0488704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0488887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0489239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0489407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0489766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0489990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0490346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0490496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0491023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0491209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0491601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0491882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0492161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0492337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0492590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw6kxjw08
2022-11-23T02:55:37.0492858Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw6kxjw08/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0493094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzdyxk58z
2022-11-23T02:55:37.0493364Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzdyxk58z/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0493673Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnkddmoac
2022-11-23T02:55:37.0494028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnkddmoac/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0494273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp67ebzcwx
2022-11-23T02:55:37.0494528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp67ebzcwx/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0494748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0494963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0495184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0495385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0495527Z fi_getinfo: -61
2022-11-23T02:55:37.0495654Z fi_getinfo: -61
2022-11-23T02:55:37.0495777Z fi_getinfo: -61
2022-11-23T02:55:37.0495901Z fi_getinfo: -61
2022-11-23T02:55:37.0495989Z ok (7.951s)
2022-11-23T02:55:37.0496008Z 
2022-11-23T02:55:37.0496259Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0496349Z Ran 1 test in 7.951s
2022-11-23T02:55:37.0496415Z 
2022-11-23T02:55:37.0496507Z OK
2022-11-23T02:55:37.0496525Z 
2022-11-23T02:55:37.0496639Z Generating XML reports...
2022-11-23T02:55:37.0497677Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024040.xml
2022-11-23T02:55:37.0498044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0498212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0498588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0498777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0499010Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9d52g3rh
2022-11-23T02:55:37.0499277Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9d52g3rh/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0499297Z 
2022-11-23T02:55:37.0499406Z Running tests...
2022-11-23T02:55:37.0499753Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0500264Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0500572Z   test_device_map_gpu_default_to_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0501289Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/80008 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.713s)
2022-11-23T02:55:37.0501311Z 
2022-11-23T02:55:37.0501561Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0501667Z Ran 1 test in 1.713s
2022-11-23T02:55:37.0501689Z 
2022-11-23T02:55:37.0501788Z OK (skipped=1)
2022-11-23T02:55:37.0501806Z 
2022-11-23T02:55:37.0501910Z Generating XML reports...
2022-11-23T02:55:37.0502436Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024051.xml
2022-11-23T02:55:37.0502784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0502949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0503304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0503483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0503724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdyvykoij
2022-11-23T02:55:37.0503986Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdyvykoij/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0504005Z 
2022-11-23T02:55:37.0504107Z Running tests...
2022-11-23T02:55:37.0504343Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0504681Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0504968Z   test_device_map_gpu_mixed_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0505176Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88672
2022-11-23T02:55:37.0505378Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88673
2022-11-23T02:55:37.0505577Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 88674
2022-11-23T02:55:37.0505778Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 88675
2022-11-23T02:55:37.0506208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0506370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0506741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0506921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0507271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0507432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0507785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0507963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0508310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0508476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0508871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0509048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0509391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0509553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0509914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0510085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0510325Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpst6stk0e
2022-11-23T02:55:37.0510585Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpst6stk0e/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0510818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppd1dvbhh
2022-11-23T02:55:37.0511079Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppd1dvbhh/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0511501Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpttd9297x
2022-11-23T02:55:37.0511761Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpttd9297x/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0512140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe0jua7_w
2022-11-23T02:55:37.0512280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe0jua7_w/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0512508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0512739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0512963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0513177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0513325Z fi_getinfo: -61
2022-11-23T02:55:37.0513458Z fi_getinfo: -61
2022-11-23T02:55:37.0513592Z fi_getinfo: -61
2022-11-23T02:55:37.0513729Z fi_getinfo: -61
2022-11-23T02:55:37.0513823Z ok (10.747s)
2022-11-23T02:55:37.0513843Z 
2022-11-23T02:55:37.0514254Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0514345Z Ran 1 test in 10.748s
2022-11-23T02:55:37.0514382Z 
2022-11-23T02:55:37.0514451Z OK
2022-11-23T02:55:37.0514469Z 
2022-11-23T02:55:37.0514585Z Generating XML reports...
2022-11-23T02:55:37.0515292Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024055.xml
2022-11-23T02:55:37.0515708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0515887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0516273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0516460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0516714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2rz3dsex
2022-11-23T02:55:37.0516966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2rz3dsex/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0516986Z 
2022-11-23T02:55:37.0517089Z Running tests...
2022-11-23T02:55:37.0517346Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0517861Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0518152Z   test_device_map_gpu_mixed_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0518414Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89183
2022-11-23T02:55:37.0518625Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89184
2022-11-23T02:55:37.0518825Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 89185
2022-11-23T02:55:37.0519013Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 89186
2022-11-23T02:55:37.0519378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0519545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0519908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0520095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0520441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0520613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0520967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0521149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0521479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0521646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0522001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0522186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0522535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0522874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0523248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0523432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0523670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdj3zh30b
2022-11-23T02:55:37.0523938Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdj3zh30b/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0524192Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7fbnvh43
2022-11-23T02:55:37.0524462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7fbnvh43/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0524756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp04cmn4xm
2022-11-23T02:55:37.0525029Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp04cmn4xm/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0525278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5nclv_q8
2022-11-23T02:55:37.0525536Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5nclv_q8/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0525920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0526123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0526332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0526553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0526696Z fi_getinfo: -61
2022-11-23T02:55:37.0526822Z fi_getinfo: -61
2022-11-23T02:55:37.0526950Z fi_getinfo: -61
2022-11-23T02:55:37.0527129Z fi_getinfo: -61
2022-11-23T02:55:37.0527206Z ok (10.520s)
2022-11-23T02:55:37.0527242Z 
2022-11-23T02:55:37.0527481Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0527592Z Ran 1 test in 10.521s
2022-11-23T02:55:37.0527610Z 
2022-11-23T02:55:37.0527689Z OK
2022-11-23T02:55:37.0527708Z 
2022-11-23T02:55:37.0527828Z Generating XML reports...
2022-11-23T02:55:37.0528538Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024109.xml
2022-11-23T02:55:37.0528910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0529087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0529465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0529642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0529899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpflv1o8et
2022-11-23T02:55:37.0530165Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpflv1o8et/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0530186Z 
2022-11-23T02:55:37.0530286Z Running tests...
2022-11-23T02:55:37.0530548Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0530901Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0531348Z   test_device_map_gpu_mixed_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0531560Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89694
2022-11-23T02:55:37.0531757Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89695
2022-11-23T02:55:37.0531963Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 89696
2022-11-23T02:55:37.0532169Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 89697
2022-11-23T02:55:37.0532529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0532702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0533068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0533254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0533604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0533769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0534340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0534539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0534902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0535072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0535440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0535627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0535982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0536150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0536508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0536904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0537169Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdj_etwyf
2022-11-23T02:55:37.0537437Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdj_etwyf/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0537687Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnt18g_z6
2022-11-23T02:55:37.0537949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnt18g_z6/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0538199Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpks1h_atu
2022-11-23T02:55:37.0538460Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpks1h_atu/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0538712Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvuda3n34
2022-11-23T02:55:37.0538959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvuda3n34/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0539188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0539404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0539623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0539844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0540139Z fi_getinfo: -61
2022-11-23T02:55:37.0540379Z fi_getinfo: -61
2022-11-23T02:55:37.0540565Z fi_getinfo: -61
2022-11-23T02:55:37.0540686Z fi_getinfo: -61
2022-11-23T02:55:37.0540778Z ok (10.573s)
2022-11-23T02:55:37.0540809Z 
2022-11-23T02:55:37.0541051Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0541159Z Ran 1 test in 10.573s
2022-11-23T02:55:37.0541178Z 
2022-11-23T02:55:37.0541263Z OK
2022-11-23T02:55:37.0541284Z 
2022-11-23T02:55:37.0541400Z Generating XML reports...
2022-11-23T02:55:37.0541942Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024122.xml
2022-11-23T02:55:37.0542314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0542476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0542850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0543036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0543290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbvsrz0jj
2022-11-23T02:55:37.0543628Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbvsrz0jj/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0543650Z 
2022-11-23T02:55:37.0543764Z Running tests...
2022-11-23T02:55:37.0544029Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0544386Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0544669Z   test_device_map_gpu_mixed_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0544887Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90205
2022-11-23T02:55:37.0545101Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90206
2022-11-23T02:55:37.0545472Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 90207
2022-11-23T02:55:37.0545676Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 90208
2022-11-23T02:55:37.0546223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0546504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0547043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0547180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0547578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0547785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0548181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0548397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0548792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0548945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0549525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0549800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0550203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0550411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0550937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0551042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0551335Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyley1op5
2022-11-23T02:55:37.0551646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyley1op5/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0551897Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ukg389c
2022-11-23T02:55:37.0552237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ukg389c/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0552563Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpilin7wur
2022-11-23T02:55:37.0552880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpilin7wur/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0553172Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4y2jgjtr
2022-11-23T02:55:37.0553464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4y2jgjtr/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0553730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0554049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0554330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0554530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0554718Z fi_getinfo: -61
2022-11-23T02:55:37.0554932Z fi_getinfo: -61
2022-11-23T02:55:37.0555104Z fi_getinfo: -61
2022-11-23T02:55:37.0555274Z fi_getinfo: -61
2022-11-23T02:55:37.0555408Z ok (10.510s)
2022-11-23T02:55:37.0555430Z 
2022-11-23T02:55:37.0555739Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0555836Z Ran 1 test in 10.511s
2022-11-23T02:55:37.0555855Z 
2022-11-23T02:55:37.0555983Z OK
2022-11-23T02:55:37.0556002Z 
2022-11-23T02:55:37.0556194Z Generating XML reports...
2022-11-23T02:55:37.0556791Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024136.xml
2022-11-23T02:55:37.0557391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0557648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0558235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0558471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0558708Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2iw5p2bv
2022-11-23T02:55:37.0559014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2iw5p2bv/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0559034Z 
2022-11-23T02:55:37.0559178Z Running tests...
2022-11-23T02:55:37.0559481Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0559882Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0560261Z   test_device_map_gpu_mixed_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0560524Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90716
2022-11-23T02:55:37.0560788Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90717
2022-11-23T02:55:37.0561038Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 90718
2022-11-23T02:55:37.0561239Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 90719
2022-11-23T02:55:37.0561717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0562123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0562577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0562749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0563183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0563397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0563796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0564038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0564353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0564558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0564950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0565217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0565620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0565862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0566285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0566480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0566711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6jxuhd5n
2022-11-23T02:55:37.0567005Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6jxuhd5n/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0567315Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp02y5gfkr
2022-11-23T02:55:37.0567611Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp02y5gfkr/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0567896Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsu0n9fdj
2022-11-23T02:55:37.0568233Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsu0n9fdj/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0568546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ckwr7tg
2022-11-23T02:55:37.0569010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ckwr7tg/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0569276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0569484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0569743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0570002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0570201Z fi_getinfo: -61
2022-11-23T02:55:37.0570374Z fi_getinfo: -61
2022-11-23T02:55:37.0570644Z fi_getinfo: -61
2022-11-23T02:55:37.0570907Z fi_getinfo: -61
2022-11-23T02:55:37.0570937Z ok (10.549s)
2022-11-23T02:55:37.0570958Z 
2022-11-23T02:55:37.0571260Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0571404Z Ran 1 test in 10.549s
2022-11-23T02:55:37.0571423Z 
2022-11-23T02:55:37.0571585Z OK
2022-11-23T02:55:37.0571605Z 
2022-11-23T02:55:37.0571773Z Generating XML reports...
2022-11-23T02:55:37.0572357Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024149.xml
2022-11-23T02:55:37.0572767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0572976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0573341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0573610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0573897Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn12sgk21
2022-11-23T02:55:37.0574200Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn12sgk21/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0574220Z 
2022-11-23T02:55:37.0574528Z Running tests...
2022-11-23T02:55:37.0574825Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0575208Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0575531Z   test_device_map_gpu_mixed_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0575775Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91235
2022-11-23T02:55:37.0576036Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91236
2022-11-23T02:55:37.0576327Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 91237
2022-11-23T02:55:37.0576580Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 91238
2022-11-23T02:55:37.0577462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0577783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0578099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0578326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0578735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0578891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0579311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0579676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0580089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0580298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0580862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0581078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0581692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0581901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0582271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0582509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0582840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnf_aqqn7
2022-11-23T02:55:37.0583183Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnf_aqqn7/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0583473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoe_33q3c
2022-11-23T02:55:37.0583775Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoe_33q3c/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0584062Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp14yx6dmg
2022-11-23T02:55:37.0584366Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp14yx6dmg/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0584608Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfgpocx6l
2022-11-23T02:55:37.0584921Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfgpocx6l/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0585192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0585481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0585749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0586008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0586194Z fi_getinfo: -61
2022-11-23T02:55:37.0586368Z fi_getinfo: -61
2022-11-23T02:55:37.0586487Z fi_getinfo: -61
2022-11-23T02:55:37.0586666Z fi_getinfo: -61
2022-11-23T02:55:37.0586798Z ok (10.540s)
2022-11-23T02:55:37.0586820Z 
2022-11-23T02:55:37.0587122Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0587404Z Ran 1 test in 10.540s
2022-11-23T02:55:37.0587425Z 
2022-11-23T02:55:37.0587556Z OK
2022-11-23T02:55:37.0587579Z 
2022-11-23T02:55:37.0587741Z Generating XML reports...
2022-11-23T02:55:37.0588487Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024202.xml
2022-11-23T02:55:37.0588836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0589045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0589448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0589666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0589945Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaq9m_n7t
2022-11-23T02:55:37.0590279Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaq9m_n7t/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0590346Z 
2022-11-23T02:55:37.0590493Z Running tests...
2022-11-23T02:55:37.0590799Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0591359Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0591648Z   test_device_map_gpu_mixed_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0591902Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91754
2022-11-23T02:55:37.0592254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91755
2022-11-23T02:55:37.0592407Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 91756
2022-11-23T02:55:37.0592690Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 91757
2022-11-23T02:55:37.0593141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0593370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0593791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0594025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0594516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0594717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0595120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0595338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0595730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0595979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0596386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0596603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0612027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0612319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0612694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0612887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0613149Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_73yzz61
2022-11-23T02:55:37.0613528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_73yzz61/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0613799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7at71naq
2022-11-23T02:55:37.0614068Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7at71naq/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0614323Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc95jux0u
2022-11-23T02:55:37.0614747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc95jux0u/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0614976Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb8udtyx6
2022-11-23T02:55:37.0615411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb8udtyx6/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0615642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0615870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0616157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0616386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0616549Z fi_getinfo: -61
2022-11-23T02:55:37.0616968Z fi_getinfo: -61
2022-11-23T02:55:37.0617113Z fi_getinfo: -61
2022-11-23T02:55:37.0617258Z fi_getinfo: -61
2022-11-23T02:55:37.0617361Z ok (10.644s)
2022-11-23T02:55:37.0617396Z 
2022-11-23T02:55:37.0617658Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0617768Z Ran 1 test in 10.645s
2022-11-23T02:55:37.0617913Z 
2022-11-23T02:55:37.0617927Z OK
2022-11-23T02:55:37.0617927Z 
2022-11-23T02:55:37.0618115Z Generating XML reports...
2022-11-23T02:55:37.0618570Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024216.xml
2022-11-23T02:55:37.0618952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0619135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0619524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0619716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0619977Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9nh5save
2022-11-23T02:55:37.0620396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9nh5save/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0620416Z 
2022-11-23T02:55:37.0620527Z Running tests...
2022-11-23T02:55:37.0620788Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0621123Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0621421Z   test_device_map_gpu_mixed_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0621641Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92273
2022-11-23T02:55:37.0621858Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92274
2022-11-23T02:55:37.0622070Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 92275
2022-11-23T02:55:37.0622280Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 92276
2022-11-23T02:55:37.0622642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0622813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0623450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0623636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0624017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0624193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0624568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0624753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0625117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0625294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0625666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0625838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0626197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0626439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0626811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0627002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0627262Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpktaorijl
2022-11-23T02:55:37.0627536Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpktaorijl/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0627795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyb4yiwkv
2022-11-23T02:55:37.0628071Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyb4yiwkv/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0628309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps7kb31ed
2022-11-23T02:55:37.0628581Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps7kb31ed/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0628849Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo6k2bxkv
2022-11-23T02:55:37.0629113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo6k2bxkv/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0629339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0629672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0629795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0630020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0630154Z fi_getinfo: -61
2022-11-23T02:55:37.0630294Z fi_getinfo: -61
2022-11-23T02:55:37.0630439Z fi_getinfo: -61
2022-11-23T02:55:37.0630582Z fi_getinfo: -61
2022-11-23T02:55:37.0630683Z ok (10.611s)
2022-11-23T02:55:37.0630703Z 
2022-11-23T02:55:37.0630967Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0631081Z Ran 1 test in 10.611s
2022-11-23T02:55:37.0631100Z 
2022-11-23T02:55:37.0631174Z OK
2022-11-23T02:55:37.0631212Z 
2022-11-23T02:55:37.0631317Z Generating XML reports...
2022-11-23T02:55:37.0632014Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024229.xml
2022-11-23T02:55:37.0632378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0632547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0633084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0633176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0633596Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpss6f9hjq
2022-11-23T02:55:37.0633864Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpss6f9hjq/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0633884Z 
2022-11-23T02:55:37.0633974Z Running tests...
2022-11-23T02:55:37.0634245Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0634606Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0634919Z   test_device_map_gpu_mixed_self_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0635146Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92792
2022-11-23T02:55:37.0635372Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92793
2022-11-23T02:55:37.0635638Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 92794
2022-11-23T02:55:37.0635859Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 92795
2022-11-23T02:55:37.0636243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0636403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0636784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0636977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0637345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0637527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0637905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0638095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0638452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0638608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0638979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0639159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0639670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0639838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0640207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0640393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0640816Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_8nlmn37
2022-11-23T02:55:37.0641085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_8nlmn37/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0641336Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5btvbl1v
2022-11-23T02:55:37.0641596Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5btvbl1v/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0641977Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2mchz90p
2022-11-23T02:55:37.0642252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2mchz90p/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0642556Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0f830eaj
2022-11-23T02:55:37.0642838Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0f830eaj/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0643081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0643298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0643508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0643731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0643885Z fi_getinfo: -61
2022-11-23T02:55:37.0644021Z fi_getinfo: -61
2022-11-23T02:55:37.0644155Z fi_getinfo: -61
2022-11-23T02:55:37.0644286Z fi_getinfo: -61
2022-11-23T02:55:37.0644383Z ok (10.659s)
2022-11-23T02:55:37.0644404Z 
2022-11-23T02:55:37.0644653Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0644770Z Ran 1 test in 10.659s
2022-11-23T02:55:37.0644789Z 
2022-11-23T02:55:37.0644973Z OK
2022-11-23T02:55:37.0644992Z 
2022-11-23T02:55:37.0645116Z Generating XML reports...
2022-11-23T02:55:37.0645663Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024243.xml
2022-11-23T02:55:37.0646032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0646207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0646584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0646775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0647165Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp65wbp8js
2022-11-23T02:55:37.0647437Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp65wbp8js/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0647453Z 
2022-11-23T02:55:37.0647842Z Running tests...
2022-11-23T02:55:37.0648109Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0648348Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0648657Z   test_device_map_gpu_mixed_self_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0648874Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93303
2022-11-23T02:55:37.0649089Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93304
2022-11-23T02:55:37.0649302Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 93305
2022-11-23T02:55:37.0649498Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 93306
2022-11-23T02:55:37.0649875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0650053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0650430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0650620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0650981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0651150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0651521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0651694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0652117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0652300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0652672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0652855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0653217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0653387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0653871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0653937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0654180Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdz8yj7fn
2022-11-23T02:55:37.0654451Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdz8yj7fn/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0654754Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp820m3_zy
2022-11-23T02:55:37.0655022Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp820m3_zy/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0655274Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnaywsrf_
2022-11-23T02:55:37.0655542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnaywsrf_/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0655792Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprxr384i0
2022-11-23T02:55:37.0656051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprxr384i0/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0656267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0656491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0657095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0657219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0657524Z fi_getinfo: -61
2022-11-23T02:55:37.0657654Z fi_getinfo: -61
2022-11-23T02:55:37.0657782Z fi_getinfo: -61
2022-11-23T02:55:37.0657911Z fi_getinfo: -61
2022-11-23T02:55:37.0657991Z ok (10.606s)
2022-11-23T02:55:37.0658011Z 
2022-11-23T02:55:37.0658437Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0658550Z Ran 1 test in 10.606s
2022-11-23T02:55:37.0658569Z 
2022-11-23T02:55:37.0658659Z OK
2022-11-23T02:55:37.0658678Z 
2022-11-23T02:55:37.0658802Z Generating XML reports...
2022-11-23T02:55:37.0659354Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024256.xml
2022-11-23T02:55:37.0659731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0659906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0660272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0660459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0660714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppfg_ch0u
2022-11-23T02:55:37.0660985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppfg_ch0u/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0661005Z 
2022-11-23T02:55:37.0661115Z Running tests...
2022-11-23T02:55:37.0661425Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0661864Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0662187Z   test_device_map_gpu_mixed_self_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0662557Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93814
2022-11-23T02:55:37.0662752Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93815
2022-11-23T02:55:37.0662958Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 93816
2022-11-23T02:55:37.0663285Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 93817
2022-11-23T02:55:37.0663608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0663698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0664066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0664370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0664664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0664813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0665223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0665354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0665710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0665877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0666302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0666424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0666781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0666950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0667293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0667474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0667722Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuo9basga
2022-11-23T02:55:37.0667984Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuo9basga/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0668226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4melk8tk
2022-11-23T02:55:37.0668486Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4melk8tk/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0668756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9f2rdq4d
2022-11-23T02:55:37.0668988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9f2rdq4d/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0669504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpshy46vk6
2022-11-23T02:55:37.0669650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpshy46vk6/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0669876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0670103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0670328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0670595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0670748Z fi_getinfo: -61
2022-11-23T02:55:37.0670886Z fi_getinfo: -61
2022-11-23T02:55:37.0671006Z fi_getinfo: -61
2022-11-23T02:55:37.0671223Z fi_getinfo: -61
2022-11-23T02:55:37.0671240Z ok (10.551s)
2022-11-23T02:55:37.0671260Z 
2022-11-23T02:55:37.0671524Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0671634Z Ran 1 test in 10.552s
2022-11-23T02:55:37.0671653Z 
2022-11-23T02:55:37.0671742Z OK
2022-11-23T02:55:37.0671761Z 
2022-11-23T02:55:37.0671884Z Generating XML reports...
2022-11-23T02:55:37.0672569Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024310.xml
2022-11-23T02:55:37.0672913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0673083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0673449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0673687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0673933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk7orj40u
2022-11-23T02:55:37.0674191Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk7orj40u/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0674210Z 
2022-11-23T02:55:37.0674314Z Running tests...
2022-11-23T02:55:37.0674574Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0674920Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0675301Z   test_device_map_gpu_mixed_self_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0675421Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94325
2022-11-23T02:55:37.0675627Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94326
2022-11-23T02:55:37.0675960Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 94327
2022-11-23T02:55:37.0676112Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 94328
2022-11-23T02:55:37.0676401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0676572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0676937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0677104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0677450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0677619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0678152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0678348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0678704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0678872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0679240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0679427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0679776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0679996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0680371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0680561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0680816Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv1qf9pm7
2022-11-23T02:55:37.0681260Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv1qf9pm7/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0681523Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpryf_fky6
2022-11-23T02:55:37.0681952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpryf_fky6/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0682190Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm98r0muo
2022-11-23T02:55:37.0682453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm98r0muo/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0682710Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnk2g2kn_
2022-11-23T02:55:37.0683026Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnk2g2kn_/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0683254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0683477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0683701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0683922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0684071Z fi_getinfo: -61
2022-11-23T02:55:37.0684192Z fi_getinfo: -61
2022-11-23T02:55:37.0684327Z fi_getinfo: -61
2022-11-23T02:55:37.0684463Z fi_getinfo: -61
2022-11-23T02:55:37.0684561Z ok (10.641s)
2022-11-23T02:55:37.0684581Z 
2022-11-23T02:55:37.0684849Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0684956Z Ran 1 test in 10.641s
2022-11-23T02:55:37.0684978Z 
2022-11-23T02:55:37.0685068Z OK
2022-11-23T02:55:37.0685087Z 
2022-11-23T02:55:37.0685193Z Generating XML reports...
2022-11-23T02:55:37.0685740Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024323.xml
2022-11-23T02:55:37.0686113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0686286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0686662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0686851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0687110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprqdicmpv
2022-11-23T02:55:37.0687380Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprqdicmpv/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0687402Z 
2022-11-23T02:55:37.0687509Z Running tests...
2022-11-23T02:55:37.0687758Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0688268Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0688591Z   test_device_map_gpu_mixed_self_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0688803Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94836
2022-11-23T02:55:37.0689013Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94837
2022-11-23T02:55:37.0689347Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 94838
2022-11-23T02:55:37.0689470Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 94839
2022-11-23T02:55:37.0689841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0689997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0690348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0690515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0690877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0691058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0691586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0691774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0692140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0692363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0692721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0692908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0693275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0693443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0693807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0694034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0694247Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa4o195rb
2022-11-23T02:55:37.0694666Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa4o195rb/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0694900Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmxa0e5c1
2022-11-23T02:55:37.0695157Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmxa0e5c1/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0695398Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6h3_jm39
2022-11-23T02:55:37.0695654Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6h3_jm39/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0695892Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe3d_ji58
2022-11-23T02:55:37.0696142Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe3d_ji58/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0696366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0696585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0697251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0697467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0697643Z fi_getinfo: -61
2022-11-23T02:55:37.0697771Z fi_getinfo: -61
2022-11-23T02:55:37.0697900Z fi_getinfo: -61
2022-11-23T02:55:37.0698102Z fi_getinfo: -61
2022-11-23T02:55:37.0698131Z ok (10.550s)
2022-11-23T02:55:37.0698150Z 
2022-11-23T02:55:37.0698414Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0698510Z Ran 1 test in 10.551s
2022-11-23T02:55:37.0698529Z 
2022-11-23T02:55:37.0698616Z OK
2022-11-23T02:55:37.0698636Z 
2022-11-23T02:55:37.0698760Z Generating XML reports...
2022-11-23T02:55:37.0699385Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024337.xml
2022-11-23T02:55:37.0699777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0699953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0700330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0700522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0700913Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzxgcexun
2022-11-23T02:55:37.0701174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzxgcexun/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0701193Z 
2022-11-23T02:55:37.0701294Z Running tests...
2022-11-23T02:55:37.0701549Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0701898Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0702262Z   test_device_map_gpu_mixed_self_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0702473Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95347
2022-11-23T02:55:37.0702684Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95348
2022-11-23T02:55:37.0702892Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 95349
2022-11-23T02:55:37.0703078Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 95350
2022-11-23T02:55:37.0703510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0703612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0703982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0704169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0704522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0704690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0705055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0705238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0705571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0705733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0706095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0706279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0706636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0706799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0707155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0707337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0707569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptzwxnrtd
2022-11-23T02:55:37.0707834Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptzwxnrtd/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0708078Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpug64z86z
2022-11-23T02:55:37.0708382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpug64z86z/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0708633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpza0bl1xy
2022-11-23T02:55:37.0708888Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpza0bl1xy/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0709123Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ns7uk98
2022-11-23T02:55:37.0709375Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ns7uk98/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0709601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0709803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0710020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0710239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0710382Z fi_getinfo: -61
2022-11-23T02:55:37.0710565Z fi_getinfo: -61
2022-11-23T02:55:37.0710694Z fi_getinfo: -61
2022-11-23T02:55:37.0710824Z fi_getinfo: -61
2022-11-23T02:55:37.0710904Z ok (10.656s)
2022-11-23T02:55:37.0710923Z 
2022-11-23T02:55:37.0711176Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0711283Z Ran 1 test in 10.656s
2022-11-23T02:55:37.0711302Z 
2022-11-23T02:55:37.0711389Z OK
2022-11-23T02:55:37.0711407Z 
2022-11-23T02:55:37.0711525Z Generating XML reports...
2022-11-23T02:55:37.0712229Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024350.xml
2022-11-23T02:55:37.0712706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0712911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0713140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0713335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0713588Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvhzb3nxm
2022-11-23T02:55:37.0713855Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvhzb3nxm/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0713875Z 
2022-11-23T02:55:37.0713982Z Running tests...
2022-11-23T02:55:37.0714247Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0714601Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0715061Z   test_device_map_gpu_mixed_self_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0715279Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95858
2022-11-23T02:55:37.0715685Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95859
2022-11-23T02:55:37.0715903Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 95860
2022-11-23T02:55:37.0716111Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 95861
2022-11-23T02:55:37.0716476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0716648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0717025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0717212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0717639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0717803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0718287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0718347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0718866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0719048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0719406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0719585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0719928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0720095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0720440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0720671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0720957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgegny8jf
2022-11-23T02:55:37.0721217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgegny8jf/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0721464Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyx99lvjp
2022-11-23T02:55:37.0721723Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyx99lvjp/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0721965Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn716o6y7
2022-11-23T02:55:37.0722221Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn716o6y7/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0722463Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3d_0wgg9
2022-11-23T02:55:37.0722700Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3d_0wgg9/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0722922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0723141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0723358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0723751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0723908Z fi_getinfo: -61
2022-11-23T02:55:37.0724033Z fi_getinfo: -61
2022-11-23T02:55:37.0724151Z fi_getinfo: -61
2022-11-23T02:55:37.0724285Z fi_getinfo: -61
2022-11-23T02:55:37.0724383Z ok (10.524s)
2022-11-23T02:55:37.0724403Z 
2022-11-23T02:55:37.0724671Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0724896Z Ran 1 test in 10.524s
2022-11-23T02:55:37.0724896Z 
2022-11-23T02:55:37.0724896Z OK
2022-11-23T02:55:37.0724911Z 
2022-11-23T02:55:37.0725031Z Generating XML reports...
2022-11-23T02:55:37.0725578Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024404.xml
2022-11-23T02:55:37.0725931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0726105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0726480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0726820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0727119Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplx7u4pkg
2022-11-23T02:55:37.0727385Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplx7u4pkg/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0727407Z 
2022-11-23T02:55:37.0727509Z Running tests...
2022-11-23T02:55:37.0727766Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0728110Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0728391Z   test_device_map_gpu_mixed_self_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0728598Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96369
2022-11-23T02:55:37.0728804Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96370
2022-11-23T02:55:37.0729197Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 96371
2022-11-23T02:55:37.0729396Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 96372
2022-11-23T02:55:37.0729770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0729990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0730368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0730542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0730904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0731075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0731444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0731635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0732146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0732314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0732667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0732846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0733180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0733341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0733696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0733895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0734121Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5iotiv2w
2022-11-23T02:55:37.0734387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5iotiv2w/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0734808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp92tqpnhw
2022-11-23T02:55:37.0735074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp92tqpnhw/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0735328Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbkzt9kpt
2022-11-23T02:55:37.0735580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbkzt9kpt/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0735827Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprs07p6qg
2022-11-23T02:55:37.0736094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprs07p6qg/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0736364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0736601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0737210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0737491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0737662Z fi_getinfo: -61
2022-11-23T02:55:37.0737701Z fi_getinfo: -61
2022-11-23T02:55:37.0737791Z fi_getinfo: -61
2022-11-23T02:55:37.0737928Z fi_getinfo: -61
2022-11-23T02:55:37.0738024Z ok (10.654s)
2022-11-23T02:55:37.0738044Z 
2022-11-23T02:55:37.0738308Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0738433Z Ran 1 test in 10.654s
2022-11-23T02:55:37.0738455Z 
2022-11-23T02:55:37.0738525Z OK
2022-11-23T02:55:37.0738543Z 
2022-11-23T02:55:37.0738650Z Generating XML reports...
2022-11-23T02:55:37.0739203Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024417.xml
2022-11-23T02:55:37.0739660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0739834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0740215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0740405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0740656Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoksnbqeu
2022-11-23T02:55:37.0740918Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoksnbqeu/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0740938Z 
2022-11-23T02:55:37.0741117Z Running tests...
2022-11-23T02:55:37.0741446Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0741794Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0742095Z   test_device_map_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0742304Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96880
2022-11-23T02:55:37.0742514Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96881
2022-11-23T02:55:37.0742718Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 96882
2022-11-23T02:55:37.0742919Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 96883
2022-11-23T02:55:37.0743460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0743618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0743999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0744192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0744554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0744725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0745095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0745281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0745637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0745807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0746371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0746564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0746919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0747081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0747436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0747612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0747854Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2mv777ve
2022-11-23T02:55:37.0748097Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnswrlhtg
2022-11-23T02:55:37.0748343Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2mv777ve/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0748604Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnswrlhtg/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0748898Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplwyhcj8s
2022-11-23T02:55:37.0749156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplwyhcj8s/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0749398Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv5072y8n
2022-11-23T02:55:37.0749651Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv5072y8n/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0749870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0750268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0750492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0750698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0750847Z fi_getinfo: -61
2022-11-23T02:55:37.0750979Z fi_getinfo: -61
2022-11-23T02:55:37.0751114Z fi_getinfo: -61
2022-11-23T02:55:37.0751244Z fi_getinfo: -61
2022-11-23T02:55:37.0751342Z ok (8.028s)
2022-11-23T02:55:37.0751361Z 
2022-11-23T02:55:37.0751722Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0751722Z Ran 1 test in 8.028s
2022-11-23T02:55:37.0751751Z 
2022-11-23T02:55:37.0751825Z OK
2022-11-23T02:55:37.0751844Z 
2022-11-23T02:55:37.0751966Z Generating XML reports...
2022-11-23T02:55:37.0752512Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024430.xml
2022-11-23T02:55:37.0752881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0753058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0753588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0753942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0754190Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqhqt65yy
2022-11-23T02:55:37.0754441Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqhqt65yy/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0754461Z 
2022-11-23T02:55:37.0754567Z Running tests...
2022-11-23T02:55:37.0754827Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0755183Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0755508Z   test_device_map_gpu_non_default_to_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0755772Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97387
2022-11-23T02:55:37.0755997Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97388
2022-11-23T02:55:37.0756211Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 97389
2022-11-23T02:55:37.0756403Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 97390
2022-11-23T02:55:37.0756778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0756951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0757328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0757518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0758035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0758201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0758787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0758945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0759302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0759489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0759863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0760050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0760415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0760590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0760966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0761152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0761450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg4ai8_ll
2022-11-23T02:55:37.0761719Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg4ai8_ll/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0761971Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprnv_k7j1
2022-11-23T02:55:37.0762222Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe387uxsd
2022-11-23T02:55:37.0762485Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprnv_k7j1/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0762903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe387uxsd/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0763145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb8zmw6an
2022-11-23T02:55:37.0763399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb8zmw6an/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0763618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0763821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0764040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0764253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0764399Z fi_getinfo: -61
2022-11-23T02:55:37.0764531Z fi_getinfo: -61
2022-11-23T02:55:37.0764732Z fi_getinfo: -61
2022-11-23T02:55:37.0764788Z fi_getinfo: -61
2022-11-23T02:55:37.0764916Z ok (10.629s)
2022-11-23T02:55:37.0764952Z 
2022-11-23T02:55:37.0765287Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0765311Z Ran 1 test in 10.629s
2022-11-23T02:55:37.0765329Z 
2022-11-23T02:55:37.0765488Z OK
2022-11-23T02:55:37.0765488Z 
2022-11-23T02:55:37.0765554Z Generating XML reports...
2022-11-23T02:55:37.0766081Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024441.xml
2022-11-23T02:55:37.0766438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0766632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0766971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0767138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0767386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprudj199o
2022-11-23T02:55:37.0767704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprudj199o/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0767723Z 
2022-11-23T02:55:37.0767828Z Running tests...
2022-11-23T02:55:37.0768079Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0768422Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0768729Z   test_device_map_gpu_to_cpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0768940Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97906
2022-11-23T02:55:37.0769149Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97907
2022-11-23T02:55:37.0769341Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 97908
2022-11-23T02:55:37.0769715Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 97909
2022-11-23T02:55:37.0770098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0770275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0770654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0770841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0771200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0771370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0771726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0771914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0772273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0772446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0772954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0773132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0773484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0773649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0773993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0774221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0774475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc5maljc9
2022-11-23T02:55:37.0774736Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc5maljc9/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0774980Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5iax5v8i
2022-11-23T02:55:37.0775239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5iax5v8i/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0775477Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmj8j9e95
2022-11-23T02:55:37.0775730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmj8j9e95/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0775969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpok160os9
2022-11-23T02:55:37.0776206Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpok160os9/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0776423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0776884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0777110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0777328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0777474Z fi_getinfo: -61
2022-11-23T02:55:37.0777604Z fi_getinfo: -61
2022-11-23T02:55:37.0777730Z fi_getinfo: -61
2022-11-23T02:55:37.0777843Z fi_getinfo: -61
2022-11-23T02:55:37.0777936Z ok (8.097s)
2022-11-23T02:55:37.0777956Z 
2022-11-23T02:55:37.0778389Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0778500Z Ran 1 test in 8.098s
2022-11-23T02:55:37.0778519Z 
2022-11-23T02:55:37.0778607Z OK
2022-11-23T02:55:37.0778626Z 
2022-11-23T02:55:37.0778760Z Generating XML reports...
2022-11-23T02:55:37.0779305Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024455.xml
2022-11-23T02:55:37.0779685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0779845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0780221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0780408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0780658Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgrsjgd3m
2022-11-23T02:55:37.0780927Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgrsjgd3m/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0780946Z 
2022-11-23T02:55:37.0781052Z Running tests...
2022-11-23T02:55:37.0781365Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0781732Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0782034Z   test_device_map_gpu_to_cpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0782262Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98417
2022-11-23T02:55:37.0782479Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98418
2022-11-23T02:55:37.0782686Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 98419
2022-11-23T02:55:37.0782898Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 98420
2022-11-23T02:55:37.0783265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0783521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0783912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0784103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0784451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0784618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0784986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0785172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0785523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0785687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0786050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0786290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0786642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0786804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0787160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0787339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0787595Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn27uy8bq
2022-11-23T02:55:37.0787862Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn27uy8bq/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0788125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc0rhp06z
2022-11-23T02:55:37.0788396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc0rhp06z/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0788809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbmejz9q0
2022-11-23T02:55:37.0789048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbmejz9q0/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0789295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz4sxil_1
2022-11-23T02:55:37.0789551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz4sxil_1/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0789778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0789994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0790218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0790434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0790583Z fi_getinfo: -61
2022-11-23T02:55:37.0790699Z fi_getinfo: -61
2022-11-23T02:55:37.0790835Z fi_getinfo: -61
2022-11-23T02:55:37.0790967Z fi_getinfo: -61
2022-11-23T02:55:37.0791067Z ok (8.155s)
2022-11-23T02:55:37.0791086Z 
2022-11-23T02:55:37.0791345Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0791453Z Ran 1 test in 8.155s
2022-11-23T02:55:37.0791472Z 
2022-11-23T02:55:37.0791562Z OK
2022-11-23T02:55:37.0791581Z 
2022-11-23T02:55:37.0791700Z Generating XML reports...
2022-11-23T02:55:37.0792398Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024506.xml
2022-11-23T02:55:37.0792975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0793048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0793429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0793617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0793869Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6thco2ca
2022-11-23T02:55:37.0794145Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6thco2ca/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0794165Z 
2022-11-23T02:55:37.0794397Z Running tests...
2022-11-23T02:55:37.0794520Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0795024Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0795313Z   test_device_maps_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0795521Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98928
2022-11-23T02:55:37.0795781Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98929
2022-11-23T02:55:37.0795993Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 98930
2022-11-23T02:55:37.0796198Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 98931
2022-11-23T02:55:37.0796560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0796733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0797079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0797271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0797633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0797804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0798474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0798541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0798904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0799076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0799427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0799615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0799985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0800159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0800532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0800713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0800970Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqhe5xgxj
2022-11-23T02:55:37.0801391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqhe5xgxj/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0801640Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeq656rj1
2022-11-23T02:55:37.0801881Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeq656rj1/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0802126Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpds3q2gr1
2022-11-23T02:55:37.0802425Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpds3q2gr1/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0802678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9kw44_be
2022-11-23T02:55:37.0802930Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9kw44_be/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0803152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0803369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0803593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0803816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0803943Z fi_getinfo: -61
2022-11-23T02:55:37.0804076Z fi_getinfo: -61
2022-11-23T02:55:37.0804210Z fi_getinfo: -61
2022-11-23T02:55:37.0804341Z fi_getinfo: -61
2022-11-23T02:55:37.0804444Z ok (10.667s)
2022-11-23T02:55:37.0804464Z 
2022-11-23T02:55:37.0804729Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0804884Z Ran 1 test in 10.667s
2022-11-23T02:55:37.0804903Z 
2022-11-23T02:55:37.0804974Z OK
2022-11-23T02:55:37.0804992Z 
2022-11-23T02:55:37.0805113Z Generating XML reports...
2022-11-23T02:55:37.0805640Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024517.xml
2022-11-23T02:55:37.0805994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0806168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0806532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0806721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0806969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyenyo614
2022-11-23T02:55:37.0807214Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyenyo614/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0807253Z 
2022-11-23T02:55:37.0807342Z Running tests...
2022-11-23T02:55:37.0807603Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0807951Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0808239Z   test_device_maps_in_options (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0808449Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99447
2022-11-23T02:55:37.0808662Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99448
2022-11-23T02:55:37.0808880Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 99449
2022-11-23T02:55:37.0809087Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 99450
2022-11-23T02:55:37.0809433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0809603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0809966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0810154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0810504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0810670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0811030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0811264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0811607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0811774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0812138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0812498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0812858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0813073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0813414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0813605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0813862Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp1fbovh3
2022-11-23T02:55:37.0814167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp1fbovh3/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0814426Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpegqc4emi
2022-11-23T02:55:37.0814695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpegqc4emi/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0814944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ms1drpb
2022-11-23T02:55:37.0815359Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ms1drpb/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0815596Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnasxqdme
2022-11-23T02:55:37.0816015Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnasxqdme/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0816248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0816463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0816860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0817097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0817241Z fi_getinfo: -61
2022-11-23T02:55:37.0817375Z fi_getinfo: -61
2022-11-23T02:55:37.0817507Z fi_getinfo: -61
2022-11-23T02:55:37.0817637Z fi_getinfo: -61
2022-11-23T02:55:37.0817739Z ok (10.603s)
2022-11-23T02:55:37.0817759Z 
2022-11-23T02:55:37.0818007Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0818127Z Ran 1 test in 10.603s
2022-11-23T02:55:37.0818146Z 
2022-11-23T02:55:37.0818238Z OK
2022-11-23T02:55:37.0818257Z 
2022-11-23T02:55:37.0818388Z Generating XML reports...
2022-11-23T02:55:37.0819085Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024530.xml
2022-11-23T02:55:37.0819450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0819621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0819989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0820156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0820404Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp778nezfg
2022-11-23T02:55:37.0820663Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp778nezfg/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0820683Z 
2022-11-23T02:55:37.0820857Z Running tests...
2022-11-23T02:55:37.0821129Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0821480Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0821795Z   test_device_maps_invalid_max_local_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0822008Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99966
2022-11-23T02:55:37.0822212Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99967
2022-11-23T02:55:37.0822403Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 99968
2022-11-23T02:55:37.0822611Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 99969
2022-11-23T02:55:37.0822978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0823150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0823515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0823760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0824303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0824474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0824830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0825025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0825388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0825563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0825933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0826128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0826500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0826674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0827189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0827353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0827604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbdp27c8s
2022-11-23T02:55:37.0827865Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbdp27c8s/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0828120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8tozahur
2022-11-23T02:55:37.0828387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8tozahur/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0828634Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp89wpt5ko
2022-11-23T02:55:37.0828892Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp89wpt5ko/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0829140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuhwpyrjh
2022-11-23T02:55:37.0829567Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuhwpyrjh/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0829792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0830023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0830295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0830532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0830685Z fi_getinfo: -61
2022-11-23T02:55:37.0830823Z fi_getinfo: -61
2022-11-23T02:55:37.0830960Z fi_getinfo: -61
2022-11-23T02:55:37.0831076Z fi_getinfo: -61
2022-11-23T02:55:37.0831177Z ok (4.843s)
2022-11-23T02:55:37.0831197Z 
2022-11-23T02:55:37.0831460Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0831569Z Ran 1 test in 4.843s
2022-11-23T02:55:37.0831588Z 
2022-11-23T02:55:37.0831672Z OK
2022-11-23T02:55:37.0831691Z 
2022-11-23T02:55:37.0831817Z Generating XML reports...
2022-11-23T02:55:37.0832511Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024544.xml
2022-11-23T02:55:37.0832871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0833025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0833445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0833640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0833892Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxapjo4fk
2022-11-23T02:55:37.0834240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxapjo4fk/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0834240Z 
2022-11-23T02:55:37.0834277Z Running tests...
2022-11-23T02:55:37.0834533Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0835047Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0835381Z   test_device_maps_invalid_max_remote_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0835588Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100313
2022-11-23T02:55:37.0835805Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100314
2022-11-23T02:55:37.0836019Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 100315
2022-11-23T02:55:37.0836234Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 100316
2022-11-23T02:55:37.0836604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0836778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0837152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0837339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0837690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0837864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0838233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0838427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0838779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0838943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0839303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0839484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0839893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0840056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0840567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0840743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0840990Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu4hpas7f
2022-11-23T02:55:37.0841250Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu4hpas7f/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0841518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp83_yds6g
2022-11-23T02:55:37.0841744Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp83_yds6g/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0841988Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiynjpv2q
2022-11-23T02:55:37.0842241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiynjpv2q/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0842514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp58ouyqb1
2022-11-23T02:55:37.0842765Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp58ouyqb1/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0842985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0843199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0843414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0843804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0843951Z fi_getinfo: -61
2022-11-23T02:55:37.0844083Z fi_getinfo: -61
2022-11-23T02:55:37.0844204Z fi_getinfo: -61
2022-11-23T02:55:37.0844335Z fi_getinfo: -61
2022-11-23T02:55:37.0844444Z ok (4.700s)
2022-11-23T02:55:37.0844457Z 
2022-11-23T02:55:37.0844721Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0844827Z Ran 1 test in 4.700s
2022-11-23T02:55:37.0844847Z 
2022-11-23T02:55:37.0844932Z OK
2022-11-23T02:55:37.0844951Z 
2022-11-23T02:55:37.0845073Z Generating XML reports...
2022-11-23T02:55:37.0845604Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024551.xml
2022-11-23T02:55:37.0845973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0846144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0846611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0846714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0846960Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6pfpvzsa
2022-11-23T02:55:37.0847382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6pfpvzsa/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0847401Z 
2022-11-23T02:55:37.0847503Z Running tests...
2022-11-23T02:55:37.0847797Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0848087Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0848391Z   test_device_maps_invalid_min_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0848599Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100660
2022-11-23T02:55:37.0848809Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100661
2022-11-23T02:55:37.0849168Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 100662
2022-11-23T02:55:37.0849407Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 100663
2022-11-23T02:55:37.0849668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0849819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0850165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0850521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0850888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0851059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0851438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0851626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0852033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0852199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0852568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0852738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0853103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0853271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0853643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0853828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0854085Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiyztvazu
2022-11-23T02:55:37.0854354Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiyztvazu/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0854604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsu3_nse5
2022-11-23T02:55:37.0854866Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsu3_nse5/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0855100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp41r1krb1
2022-11-23T02:55:37.0855371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp41r1krb1/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0855608Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp02q7jspn
2022-11-23T02:55:37.0855873Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp02q7jspn/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0856104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0856335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0856554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0856951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0857089Z fi_getinfo: -61
2022-11-23T02:55:37.0857223Z fi_getinfo: -61
2022-11-23T02:55:37.0857355Z fi_getinfo: -61
2022-11-23T02:55:37.0857484Z fi_getinfo: -61
2022-11-23T02:55:37.0857580Z ok (4.704s)
2022-11-23T02:55:37.0857600Z 
2022-11-23T02:55:37.0857864Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0857971Z Ran 1 test in 4.704s
2022-11-23T02:55:37.0857990Z 
2022-11-23T02:55:37.0858133Z OK
2022-11-23T02:55:37.0858169Z 
2022-11-23T02:55:37.0858281Z Generating XML reports...
2022-11-23T02:55:37.0858830Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024559.xml
2022-11-23T02:55:37.0859198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0859372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0859748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0859937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0860187Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprq_b22dv
2022-11-23T02:55:37.0860452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprq_b22dv/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0860475Z 
2022-11-23T02:55:37.0860568Z Running tests...
2022-11-23T02:55:37.0860831Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0861248Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0861599Z   test_device_maps_many_to_one (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0861822Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100995
2022-11-23T02:55:37.0862042Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100996
2022-11-23T02:55:37.0862251Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 100997
2022-11-23T02:55:37.0862463Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 100998
2022-11-23T02:55:37.0862824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0863001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0863522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0863723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0864049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0864213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0864577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0864761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0865108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0865343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0865613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0865850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0866145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0866306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0866662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0866961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0867093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy003lu4y
2022-11-23T02:55:37.0867404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy003lu4y/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0867814Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg05b0hhr
2022-11-23T02:55:37.0868082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg05b0hhr/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0868343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpez2phpgs
2022-11-23T02:55:37.0868595Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpez2phpgs/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0868839Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiu15s6mo
2022-11-23T02:55:37.0869099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiu15s6mo/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0869327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0869551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0869763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0870034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0870180Z fi_getinfo: -61
2022-11-23T02:55:37.0870309Z fi_getinfo: -61
2022-11-23T02:55:37.0870439Z fi_getinfo: -61
2022-11-23T02:55:37.0870569Z fi_getinfo: -61
2022-11-23T02:55:37.0870666Z ok (4.840s)
2022-11-23T02:55:37.0870686Z 
2022-11-23T02:55:37.0870934Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0871045Z Ran 1 test in 4.840s
2022-11-23T02:55:37.0871064Z 
2022-11-23T02:55:37.0871152Z OK
2022-11-23T02:55:37.0871171Z 
2022-11-23T02:55:37.0871293Z Generating XML reports...
2022-11-23T02:55:37.0871969Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024607.xml
2022-11-23T02:55:37.0872209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0872386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0872759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0872944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0873178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq5s_kj76
2022-11-23T02:55:37.0873443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq5s_kj76/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0873462Z 
2022-11-23T02:55:37.0873565Z Running tests...
2022-11-23T02:55:37.0873828Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0874182Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0874492Z   test_device_maps_missing_config (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0874716Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101342
2022-11-23T02:55:37.0875084Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101343
2022-11-23T02:55:37.0875314Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 101344
2022-11-23T02:55:37.0875480Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 101345
2022-11-23T02:55:37.0875839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0876001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0876360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0876588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0876944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0877111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0877467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0877633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0877990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0878152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0878662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0878830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0879201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0879438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0879811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0879993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0880229Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6_zya43_
2022-11-23T02:55:37.0880490Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6_zya43_/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0880736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpohdwuhj9
2022-11-23T02:55:37.0880996Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpohdwuhj9/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0881246Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz5oa5n2k
2022-11-23T02:55:37.0881558Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz5oa5n2k/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0881959Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpey1nojfm
2022-11-23T02:55:37.0882214Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpey1nojfm/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0882416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0882812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0883033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0883258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0883395Z fi_getinfo: -61
2022-11-23T02:55:37.0883529Z fi_getinfo: -61
2022-11-23T02:55:37.0883664Z fi_getinfo: -61
2022-11-23T02:55:37.0883798Z fi_getinfo: -61
2022-11-23T02:55:37.0883884Z ok (6.750s)
2022-11-23T02:55:37.0883903Z 
2022-11-23T02:55:37.0884167Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0884272Z Ran 1 test in 6.750s
2022-11-23T02:55:37.0884292Z 
2022-11-23T02:55:37.0884377Z OK
2022-11-23T02:55:37.0884396Z 
2022-11-23T02:55:37.0884515Z Generating XML reports...
2022-11-23T02:55:37.0885057Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024614.xml
2022-11-23T02:55:37.0885423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0885594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0886005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0886195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0886447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8tev8rbh
2022-11-23T02:55:37.0886715Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8tev8rbh/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0886735Z 
2022-11-23T02:55:37.0886840Z Running tests...
2022-11-23T02:55:37.0887104Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0887458Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0887767Z   test_device_maps_missing_config_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0887986Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101849
2022-11-23T02:55:37.0888191Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101850
2022-11-23T02:55:37.0888398Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 101851
2022-11-23T02:55:37.0888654Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 101852
2022-11-23T02:55:37.0889175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0889340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0889696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0889906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0890224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0890373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0890730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0890913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0891256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0891416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0891768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0891944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0892466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0892633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0892988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0893168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0893423Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy8nximxn
2022-11-23T02:55:37.0893688Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy8nximxn/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0893937Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbxabcc1m
2022-11-23T02:55:37.0894201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbxabcc1m/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0894451Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfvtcqlqr
2022-11-23T02:55:37.0894744Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfvtcqlqr/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0894940Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp48ssy33h
2022-11-23T02:55:37.0895422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp48ssy33h/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0895650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0895864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0896075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0896283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0896420Z fi_getinfo: -61
2022-11-23T02:55:37.0896545Z fi_getinfo: -61
2022-11-23T02:55:37.0896776Z fi_getinfo: -61
2022-11-23T02:55:37.0897153Z fi_getinfo: -61
2022-11-23T02:55:37.0897236Z ok (6.893s)
2022-11-23T02:55:37.0897258Z 
2022-11-23T02:55:37.0897528Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0897632Z Ran 1 test in 6.893s
2022-11-23T02:55:37.0897656Z 
2022-11-23T02:55:37.0897744Z OK
2022-11-23T02:55:37.0897763Z 
2022-11-23T02:55:37.0897961Z Generating XML reports...
2022-11-23T02:55:37.0898513Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024624.xml
2022-11-23T02:55:37.0898868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0899034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0899406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0899593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0899840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkdl75akg
2022-11-23T02:55:37.0900105Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkdl75akg/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0900126Z 
2022-11-23T02:55:37.0900230Z Running tests...
2022-11-23T02:55:37.0900549Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0900852Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0901160Z   test_device_maps_missing_config_not_timeout (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0901377Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102356
2022-11-23T02:55:37.0901752Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102357
2022-11-23T02:55:37.0901959Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102358
2022-11-23T02:55:37.0902161Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 102359
2022-11-23T02:55:37.0902526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0902692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0903055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0903222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0903568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0903730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0904088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0904268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0904677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0904853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0905216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0905397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0905725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0905886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0906236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0906413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0906659Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7q5bb2te
2022-11-23T02:55:37.0906919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7q5bb2te/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0907208Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2m2ze0u5
2022-11-23T02:55:37.0907456Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2m2ze0u5/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0907681Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ynrcii4
2022-11-23T02:55:37.0907936Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ynrcii4/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0908173Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbdhp8iqv
2022-11-23T02:55:37.0908427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbdhp8iqv/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0908645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0908863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0909075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0909216Z fi_getinfo: -61
2022-11-23T02:55:37.0909330Z fi_getinfo: -61
2022-11-23T02:55:37.0909451Z fi_getinfo: -61
2022-11-23T02:55:37.0909661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0909790Z fi_getinfo: -61
2022-11-23T02:55:37.0909884Z ok (6.877s)
2022-11-23T02:55:37.0909903Z 
2022-11-23T02:55:37.0910156Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0910257Z Ran 1 test in 6.877s
2022-11-23T02:55:37.0910276Z 
2022-11-23T02:55:37.0910357Z OK
2022-11-23T02:55:37.0910375Z 
2022-11-23T02:55:37.0910477Z Generating XML reports...
2022-11-23T02:55:37.0910999Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024634.xml
2022-11-23T02:55:37.0911352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0911522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0911884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0912065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0912308Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa9j3j5ka
2022-11-23T02:55:37.0912562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa9j3j5ka/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0912581Z 
2022-11-23T02:55:37.0912851Z Running tests...
2022-11-23T02:55:37.0913101Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0913519Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0913843Z   test_device_maps_missing_config_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0914061Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102863
2022-11-23T02:55:37.0914277Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102864
2022-11-23T02:55:37.0914490Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102865
2022-11-23T02:55:37.0914700Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 102866
2022-11-23T02:55:37.0915070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0915228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0915755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0915943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0916513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0916684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0917052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0917238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0917596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0917764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0918115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0918298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0918654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0918824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0919355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0919534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0919778Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvygaioth
2022-11-23T02:55:37.0920040Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvygaioth/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0920268Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5dzx4br6
2022-11-23T02:55:37.0920527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5dzx4br6/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0920766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcwzp9b1f
2022-11-23T02:55:37.0921025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcwzp9b1f/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0921259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpclzg4jou
2022-11-23T02:55:37.0921630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpclzg4jou/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0921728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0921942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0922159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0922360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0922543Z fi_getinfo: -61
2022-11-23T02:55:37.0922682Z fi_getinfo: -61
2022-11-23T02:55:37.0922809Z fi_getinfo: -61
2022-11-23T02:55:37.0922941Z fi_getinfo: -61
2022-11-23T02:55:37.0923030Z ok (6.703s)
2022-11-23T02:55:37.0923051Z 
2022-11-23T02:55:37.0923302Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0923392Z Ran 1 test in 6.703s
2022-11-23T02:55:37.0923410Z 
2022-11-23T02:55:37.0923492Z OK
2022-11-23T02:55:37.0923511Z 
2022-11-23T02:55:37.0923626Z Generating XML reports...
2022-11-23T02:55:37.0924148Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024643.xml
2022-11-23T02:55:37.0924685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0924862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0925234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0925418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0925724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpooa60_n6
2022-11-23T02:55:37.0925968Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpooa60_n6/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0925987Z 
2022-11-23T02:55:37.0926095Z Running tests...
2022-11-23T02:55:37.0926361Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0926718Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0927041Z   test_device_maps_missing_config_remote_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0927255Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103370
2022-11-23T02:55:37.0927474Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103371
2022-11-23T02:55:37.0927690Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 103372
2022-11-23T02:55:37.0927884Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 103373
2022-11-23T02:55:37.0928256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0928427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0928803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0928989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0929597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0929843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0930213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0930407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0930750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0930919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0931283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0931467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0931826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0931996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0932403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0932746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0932975Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe_tl1kka
2022-11-23T02:55:37.0933231Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe_tl1kka/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0933474Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9cnlrfah
2022-11-23T02:55:37.0933896Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9cnlrfah/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0934234Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdonpne3x
2022-11-23T02:55:37.0934410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdonpne3x/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0934658Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpywmyjgn6
2022-11-23T02:55:37.0935023Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpywmyjgn6/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0935242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0935452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0935675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0935893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0936036Z fi_getinfo: -61
2022-11-23T02:55:37.0936163Z fi_getinfo: -61
2022-11-23T02:55:37.0936293Z fi_getinfo: -61
2022-11-23T02:55:37.0936425Z fi_getinfo: -61
2022-11-23T02:55:37.0936505Z ok (6.795s)
2022-11-23T02:55:37.0936526Z 
2022-11-23T02:55:37.0936963Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0937078Z Ran 1 test in 6.796s
2022-11-23T02:55:37.0937098Z 
2022-11-23T02:55:37.0937189Z OK
2022-11-23T02:55:37.0937208Z 
2022-11-23T02:55:37.0937326Z Generating XML reports...
2022-11-23T02:55:37.0937870Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024653.xml
2022-11-23T02:55:37.0938235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0938404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0938781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0939057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0939300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8nit6wvs
2022-11-23T02:55:37.0939587Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8nit6wvs/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0939587Z 
2022-11-23T02:55:37.0939656Z Running tests...
2022-11-23T02:55:37.0940031Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0940561Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0940937Z   test_device_maps_missing_config_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0941096Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103877
2022-11-23T02:55:37.0941360Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103878
2022-11-23T02:55:37.0941466Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 103879
2022-11-23T02:55:37.0941736Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 103880
2022-11-23T02:55:37.0942236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0942425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0942805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0942992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0943349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0943516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0943871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0944055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0944415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0944649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0945021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0945203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0945565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0945733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0946085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0946270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0946526Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp27zyd3iu
2022-11-23T02:55:37.0946791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp27zyd3iu/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0947044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptojv_1rb
2022-11-23T02:55:37.0947459Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptojv_1rb/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0947705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiykuqv9b
2022-11-23T02:55:37.0948203Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiykuqv9b/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0948377Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkox1to6_
2022-11-23T02:55:37.0948625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkox1to6_/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0948852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0949077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0949307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0949523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0949668Z fi_getinfo: -61
2022-11-23T02:55:37.0949802Z fi_getinfo: -61
2022-11-23T02:55:37.0949928Z fi_getinfo: -61
2022-11-23T02:55:37.0950043Z fi_getinfo: -61
2022-11-23T02:55:37.0950266Z ok (6.867s)
2022-11-23T02:55:37.0950266Z 
2022-11-23T02:55:37.0950556Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0950629Z Ran 1 test in 6.867s
2022-11-23T02:55:37.0950694Z 
2022-11-23T02:55:37.0950694Z OK
2022-11-23T02:55:37.0950694Z 
2022-11-23T02:55:37.0950902Z Generating XML reports...
2022-11-23T02:55:37.0951422Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024702.xml
2022-11-23T02:55:37.0951762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0951924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0952348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0952481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0952735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp49a4sap2
2022-11-23T02:55:37.0953004Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp49a4sap2/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0953019Z 
2022-11-23T02:55:37.0953124Z Running tests...
2022-11-23T02:55:37.0953381Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0953737Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0954180Z   test_device_maps_missing_config_response_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0954319Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104384
2022-11-23T02:55:37.0954535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104385
2022-11-23T02:55:37.0954752Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 104386
2022-11-23T02:55:37.0954963Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 104387
2022-11-23T02:55:37.0955331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0955502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0955882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0956075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0956421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0956590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0956960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0957145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0957504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0957677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0958047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0958232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0958735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0958884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0959401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0959581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0959832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvowl34mb
2022-11-23T02:55:37.0960103Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvowl34mb/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0960357Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ck__lwk
2022-11-23T02:55:37.0960679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ck__lwk/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0960938Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdq1nuc5e
2022-11-23T02:55:37.0961186Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdq1nuc5e/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0961431Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi73qepkg
2022-11-23T02:55:37.0961746Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi73qepkg/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0961970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0962192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0962412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0962632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0962782Z fi_getinfo: -61
2022-11-23T02:55:37.0962957Z fi_getinfo: -61
2022-11-23T02:55:37.0963091Z fi_getinfo: -61
2022-11-23T02:55:37.0963222Z fi_getinfo: -61
2022-11-23T02:55:37.0963312Z ok (6.921s)
2022-11-23T02:55:37.0963332Z 
2022-11-23T02:55:37.0963781Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0963953Z Ran 1 test in 6.921s
2022-11-23T02:55:37.0963953Z 
2022-11-23T02:55:37.0963955Z OK
2022-11-23T02:55:37.0964099Z 
2022-11-23T02:55:37.0964113Z Generating XML reports...
2022-11-23T02:55:37.0964607Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024712.xml
2022-11-23T02:55:37.0964961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0965126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0965486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0965670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0965916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpchid8ceu
2022-11-23T02:55:37.0966178Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpchid8ceu/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0966194Z 
2022-11-23T02:55:37.0966294Z Running tests...
2022-11-23T02:55:37.0966532Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0966873Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0967263Z   test_device_maps_multi_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0967371Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104891
2022-11-23T02:55:37.0967585Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104892
2022-11-23T02:55:37.0967797Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 104893
2022-11-23T02:55:37.0968003Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 104894
2022-11-23T02:55:37.0968360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0968530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0968878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0969058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0969406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0969621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0969990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0970163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0970679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0970845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0971204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0971384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0971743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0971911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0972313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0972519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0972775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfld899f_
2022-11-23T02:55:37.0973041Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfld899f_/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0973442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphxdyk6ag
2022-11-23T02:55:37.0973683Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphxdyk6ag/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0973923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp84q6dzoi
2022-11-23T02:55:37.0974183Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp84q6dzoi/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0974426Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpua100_sx
2022-11-23T02:55:37.0974683Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpua100_sx/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0974898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0975107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0975321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0975651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0975670Z fi_getinfo: -61
2022-11-23T02:55:37.0975808Z fi_getinfo: -61
2022-11-23T02:55:37.0975924Z fi_getinfo: -61
2022-11-23T02:55:37.0976050Z fi_getinfo: -61
2022-11-23T02:55:37.0976139Z ok (10.561s)
2022-11-23T02:55:37.0976283Z 
2022-11-23T02:55:37.0976446Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0976522Z Ran 1 test in 10.561s
2022-11-23T02:55:37.0976545Z 
2022-11-23T02:55:37.0976614Z OK
2022-11-23T02:55:37.0976632Z 
2022-11-23T02:55:37.0977189Z Generating XML reports...
2022-11-23T02:55:37.0977672Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024722.xml
2022-11-23T02:55:37.0978039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0978212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0978587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0978774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0979117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx_1sfgxl
2022-11-23T02:55:37.0979364Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx_1sfgxl/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0979401Z 
2022-11-23T02:55:37.0979491Z Running tests...
2022-11-23T02:55:37.0979754Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0980106Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0980408Z   test_device_maps_multi_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0980623Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105410
2022-11-23T02:55:37.0980837Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105411
2022-11-23T02:55:37.0981050Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 105412
2022-11-23T02:55:37.0981269Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 105413
2022-11-23T02:55:37.0981677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0981937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0982468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0982647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0983163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0983331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0983697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0983881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0984245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0984406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0984770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0984952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0985308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0985473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0985845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0986351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0986608Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphnu2n0vq
2022-11-23T02:55:37.0986864Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphnu2n0vq/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0987112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiax2ck0s
2022-11-23T02:55:37.0987374Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiax2ck0s/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0987619Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprx6j3cz6
2022-11-23T02:55:37.0987876Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprx6j3cz6/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0988120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp__otxnrs
2022-11-23T02:55:37.0988383Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp__otxnrs/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0988669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.0988883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.0989096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.0989470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.0989609Z fi_getinfo: -61
2022-11-23T02:55:37.0989739Z fi_getinfo: -61
2022-11-23T02:55:37.0989861Z fi_getinfo: -61
2022-11-23T02:55:37.0989987Z fi_getinfo: -61
2022-11-23T02:55:37.0990079Z ok (10.550s)
2022-11-23T02:55:37.0990098Z 
2022-11-23T02:55:37.0990431Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0990441Z Ran 1 test in 10.550s
2022-11-23T02:55:37.0990460Z 
2022-11-23T02:55:37.0990544Z OK
2022-11-23T02:55:37.0990562Z 
2022-11-23T02:55:37.0990674Z Generating XML reports...
2022-11-23T02:55:37.0991202Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024735.xml
2022-11-23T02:55:37.0991611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0991776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0992138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0992319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0992714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt1e6ntbo
2022-11-23T02:55:37.0992980Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt1e6ntbo/_remote_module_non_scriptable.py
2022-11-23T02:55:37.0993000Z 
2022-11-23T02:55:37.0993102Z Running tests...
2022-11-23T02:55:37.0993366Z ----------------------------------------------------------------------
2022-11-23T02:55:37.0993724Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.0994024Z   test_device_maps_one_to_many (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.0994240Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105921
2022-11-23T02:55:37.0994452Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105922
2022-11-23T02:55:37.0994751Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 105923
2022-11-23T02:55:37.0994861Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 105924
2022-11-23T02:55:37.0995230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0995405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0995933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0996116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0996465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0996635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0996990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0997153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0997498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0997663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0998062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0998245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0998592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.0998927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.0999295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.0999463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.0999713Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl3w_62jf
2022-11-23T02:55:37.0999972Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl3w_62jf/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1000224Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppcxi329p
2022-11-23T02:55:37.1000490Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppcxi329p/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1000790Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplkoc4voo
2022-11-23T02:55:37.1001052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplkoc4voo/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1001298Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc9tpto8a
2022-11-23T02:55:37.1001559Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc9tpto8a/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1001771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.1002133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.1002346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.1002555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.1002696Z fi_getinfo: -61
2022-11-23T02:55:37.1002791Z ok (4.626s)
2022-11-23T02:55:37.1002810Z 
2022-11-23T02:55:37.1003067Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1003167Z Ran 1 test in 4.626s
2022-11-23T02:55:37.1003187Z 
2022-11-23T02:55:37.1003256Z OK
2022-11-23T02:55:37.1003274Z 
2022-11-23T02:55:37.1003391Z Generating XML reports...
2022-11-23T02:55:37.1003916Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024749.xml
2022-11-23T02:55:37.1004266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1004432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1004800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1004982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1005227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp27ujk98o
2022-11-23T02:55:37.1005464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp27ujk98o/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1005500Z 
2022-11-23T02:55:37.1005586Z Running tests...
2022-11-23T02:55:37.1005835Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1006175Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.1006460Z   test_device_maps_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.1006671Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106253
2022-11-23T02:55:37.1006926Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106254
2022-11-23T02:55:37.1007137Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 106255
2022-11-23T02:55:37.1007347Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 106256
2022-11-23T02:55:37.1007687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1007854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1008217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1008395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1008747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1008908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1009267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1009491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1009839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1009991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1010337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1010514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1010863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1011020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1011381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1011558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1011808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgzwx13_u
2022-11-23T02:55:37.1012053Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgzwx13_u/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1012295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2j1gotsn
2022-11-23T02:55:37.1012550Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2j1gotsn/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1012793Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ghie8ag
2022-11-23T02:55:37.1013214Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ghie8ag/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1013465Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfq9yuy6k
2022-11-23T02:55:37.1013838Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfq9yuy6k/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1014035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.1014174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.1014384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.1014602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.1014747Z fi_getinfo: -61
2022-11-23T02:55:37.1014878Z fi_getinfo: -61
2022-11-23T02:55:37.1015008Z fi_getinfo: -61
2022-11-23T02:55:37.1015138Z fi_getinfo: -61
2022-11-23T02:55:37.1015230Z ok (10.559s)
2022-11-23T02:55:37.1015250Z 
2022-11-23T02:55:37.1015497Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1015655Z Ran 1 test in 10.560s
2022-11-23T02:55:37.1015676Z 
2022-11-23T02:55:37.1015766Z OK
2022-11-23T02:55:37.1015788Z 
2022-11-23T02:55:37.1016065Z Generating XML reports...
2022-11-23T02:55:37.1017065Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024756.xml
2022-11-23T02:55:37.1017391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1017503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1017875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1018050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1018303Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ntogq1s
2022-11-23T02:55:37.1018572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ntogq1s/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1018591Z 
2022-11-23T02:55:37.1018781Z Running tests...
2022-11-23T02:55:37.1019045Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1019401Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.1019860Z   test_device_maps_return_to_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.1020068Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106772
2022-11-23T02:55:37.1020278Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106773
2022-11-23T02:55:37.1020468Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 106774
2022-11-23T02:55:37.1020675Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 106775
2022-11-23T02:55:37.1021037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1021204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1021569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1021749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1022096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1022257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1022608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1022772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1023124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1023289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1023648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1023824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1024165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1024331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1024886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1025047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1025300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphgi9x51c
2022-11-23T02:55:37.1025625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphgi9x51c/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1025895Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvsedafjt
2022-11-23T02:55:37.1026162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvsedafjt/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1026414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpac4ns78_
2022-11-23T02:55:37.1026677Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpac4ns78_/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1026925Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy2dp_wbe
2022-11-23T02:55:37.1027190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy2dp_wbe/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1027401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.1027622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.1028041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.1028258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.1028399Z fi_getinfo: -61
2022-11-23T02:55:37.1028526Z fi_getinfo: -61
2022-11-23T02:55:37.1028657Z fi_getinfo: -61
2022-11-23T02:55:37.1028769Z fi_getinfo: -61
2022-11-23T02:55:37.1028862Z ok (15.764s)
2022-11-23T02:55:37.1028881Z 
2022-11-23T02:55:37.1029131Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1029237Z Ran 1 test in 15.765s
2022-11-23T02:55:37.1029256Z 
2022-11-23T02:55:37.1029341Z OK
2022-11-23T02:55:37.1029359Z 
2022-11-23T02:55:37.1029475Z Generating XML reports...
2022-11-23T02:55:37.1030171Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024810.xml
2022-11-23T02:55:37.1030541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1030715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1031075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1031260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1031511Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpodkn7sq7
2022-11-23T02:55:37.1031776Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpodkn7sq7/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1031796Z 
2022-11-23T02:55:37.1031897Z Running tests...
2022-11-23T02:55:37.1032155Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1032512Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.1032817Z   test_device_maps_return_to_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.1033177Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107307
2022-11-23T02:55:37.1033384Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107308
2022-11-23T02:55:37.1033587Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 107309
2022-11-23T02:55:37.1033792Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 107310
2022-11-23T02:55:37.1034149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1034314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1034723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1034987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1035268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1035419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1035941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1036122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1036478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1036645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1037003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1037191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1037550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1037755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1038133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1038317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1038578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz4xml9oy
2022-11-23T02:55:37.1038847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz4xml9oy/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1039252Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxmkv5bsw
2022-11-23T02:55:37.1039523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxmkv5bsw/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1039765Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqd79k6nl
2022-11-23T02:55:37.1040021Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqd79k6nl/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1040246Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxqfpzu9z
2022-11-23T02:55:37.1040501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxqfpzu9z/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1040723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.1040934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.1041153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.1041370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.1041510Z fi_getinfo: -61
2022-11-23T02:55:37.1041644Z fi_getinfo: -61
2022-11-23T02:55:37.1041761Z fi_getinfo: -61
2022-11-23T02:55:37.1042020Z fi_getinfo: -61
2022-11-23T02:55:37.1042039Z ok (15.713s)
2022-11-23T02:55:37.1042039Z 
2022-11-23T02:55:37.1042272Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1042381Z Ran 1 test in 15.713s
2022-11-23T02:55:37.1042400Z 
2022-11-23T02:55:37.1042489Z OK
2022-11-23T02:55:37.1042507Z 
2022-11-23T02:55:37.1042627Z Generating XML reports...
2022-11-23T02:55:37.1043138Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024828.xml
2022-11-23T02:55:37.1043501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1043677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1044107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1044470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1044719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3io1b2n8
2022-11-23T02:55:37.1044984Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3io1b2n8/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1045005Z 
2022-11-23T02:55:37.1045114Z Running tests...
2022-11-23T02:55:37.1045382Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1045721Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.1046030Z   test_device_maps_wrong_worker_name (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.1046247Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107826
2022-11-23T02:55:37.1046465Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107827
2022-11-23T02:55:37.1046732Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 107828
2022-11-23T02:55:37.1046945Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 107829
2022-11-23T02:55:37.1047475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1047650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1048017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1048186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1048583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1048704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1049064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1049250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1049600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1049946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1050310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1050479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1050839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1051008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1051379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1051567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1051838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps22v84_k
2022-11-23T02:55:37.1052089Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps22v84_k/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1052344Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe0luaipr
2022-11-23T02:55:37.1052743Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe0luaipr/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1052856Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmrtlkotk
2022-11-23T02:55:37.1053179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmrtlkotk/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1053533Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa3r57yac
2022-11-23T02:55:37.1053706Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa3r57yac/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1053937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.1054160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.1054390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.1054536Z fi_getinfo: -61
2022-11-23T02:55:37.1054739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.1054875Z fi_getinfo: -61
2022-11-23T02:55:37.1055013Z fi_getinfo: -61
2022-11-23T02:55:37.1055144Z fi_getinfo: -61
2022-11-23T02:55:37.1055244Z ok (4.797s)
2022-11-23T02:55:37.1055264Z 
2022-11-23T02:55:37.1055527Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1055638Z Ran 1 test in 4.797s
2022-11-23T02:55:37.1055659Z 
2022-11-23T02:55:37.1055779Z OK
2022-11-23T02:55:37.1055819Z 
2022-11-23T02:55:37.1055926Z Generating XML reports...
2022-11-23T02:55:37.1056477Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024847.xml
2022-11-23T02:55:37.1057269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1057349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1057713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1057901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1058152Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0pls3tz9
2022-11-23T02:55:37.1058417Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0pls3tz9/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1058574Z 
2022-11-23T02:55:37.1058574Z Running tests...
2022-11-23T02:55:37.1058945Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1059293Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.1059746Z   test_device_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.1059969Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108173
2022-11-23T02:55:37.1060193Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108174
2022-11-23T02:55:37.1060413Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 108175
2022-11-23T02:55:37.1060621Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 108176
2022-11-23T02:55:37.1060995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1061157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1061572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1061750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1062133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1062318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1062688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1062867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1063306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1063482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1063843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1064033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1064495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1064564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1064929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1065122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1065468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7zvug3ws
2022-11-23T02:55:37.1065649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7zvug3ws/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1065952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc_m9yf9q
2022-11-23T02:55:37.1066213Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc_m9yf9q/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1066537Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz7h6vxcq
2022-11-23T02:55:37.1066888Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz7h6vxcq/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1067129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpawa816lq
2022-11-23T02:55:37.1067392Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpawa816lq/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1067613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.1067838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.1068041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.1068262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.1068410Z fi_getinfo: -61
2022-11-23T02:55:37.1068549Z fi_getinfo: -61
2022-11-23T02:55:37.1068678Z fi_getinfo: -61
2022-11-23T02:55:37.1068805Z fi_getinfo: -61
2022-11-23T02:55:37.1068935Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:55:37.1081295Z RuntimeError('Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!\nException raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fa18406a59b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7fa184065dfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xced (0x7fa1902b113d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7fa1902b267f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf2 (0x7fa1902b3ef2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2e (0x7fa1905a8b7e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #6: <unknown function> + 0x2a0be3e (0x7fa186f20e3e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #7: <unknown function> + 0x2a0bf46 (0x7fa186f20f46 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7fa1910bac58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: <unknown function> + 0x35efc70 (0x7fa19292cc70 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: <unknown function> + 0x35f03e9 (0x7fa19292d3e9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x172 (0x7fa1910f4e62 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: <unknown function> + 0x2ff562 (0x7fa19c01b562 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #13: <unknown function> + 0x2ff956 (0x7fa19c01b956 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #14: <unknown function> + 0x1ddc68 (0x55ca7eed5c68 in /opt/conda/bin/python)\nframe #15: <unknown function> + 0x199499 (0x55ca7ee91499 in /opt/conda/bin/python)\nframe #16: <unknown function> + 0x1995fa (0x55ca7ee915fa in /opt/conda/bin/python)\nframe #17: PyNumber_Add + 0x41 (0x55ca7ee3d4b1 in /opt/conda/bin/python)\nframe #18: _PyEval_EvalFrameDefault + 0x1008 (0x55ca7eeda098 in /opt/conda/bin/python)\nframe #19: <unknown function> + 0x18f742 (0x55ca7ee87742 in /opt/conda/bin/python)\nframe #20: _PyObject_Call + 0x20a (0x55ca7ee3ffaa in /opt/conda/bin/python)\nframe #21: _PyEval_EvalFrameDefault + 0x26e4 (0x55ca7eedb774 in /opt/conda/bin/python)\nframe #22: <unknown function> + 0x18f742 (0x55ca7ee87742 in /opt/conda/bin/python)\nframe #23: _PyObject_Call + 0x20a (0x55ca7ee3ffaa in /opt/conda/bin/python)\nframe #24: <unknown function> + 0xaa8dba (0x7fa19c7c4dba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7fa19c7c2ffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7fa19c7c62d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7fa19c7c7b16 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7fa193d685cc in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7fa19c7c60c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #30: <unknown function> + 0x4a24a53 (0x7fa193d61a53 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7fa193d625e8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7fa193d5c8e7 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #33: <unknown function> + 0x4a545d2 (0x7fa193d915d2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7fa18405890b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #35: <unknown function> + 0xdbbf4 (0x7fa1b3ebebf4 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #36: <unknown function> + 0x76db (0x7fa1d45126db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #37: clone + 0x3f (0x7fa1d423b61f in /lib/x86_64-linux-gnu/libc.so.6)\n')
2022-11-23T02:55:37.1081602Z Traceback (most recent call last):
2022-11-23T02:55:37.1081972Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:55:37.1082160Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:55:37.1082725Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 5954, in _gpu_add_wrong_gpus
2022-11-23T02:55:37.1082889Z     return x.cpu() + y.cuda()
2022-11-23T02:55:37.1083128Z RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
2022-11-23T02:55:37.1083575Z Exception raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):
2022-11-23T02:55:37.1084122Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fa18406a59b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:55:37.1084743Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7fa184065dfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:55:37.1085245Z frame #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xced (0x7fa1902b113d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1085709Z frame #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7fa1902b267f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1086437Z frame #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf2 (0x7fa1902b3ef2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1087117Z frame #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2e (0x7fa1905a8b7e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1087505Z frame #6: <unknown function> + 0x2a0be3e (0x7fa186f20e3e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:55:37.1087885Z frame #7: <unknown function> + 0x2a0bf46 (0x7fa186f20f46 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:55:37.1088441Z frame #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7fa1910bac58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1088825Z frame #9: <unknown function> + 0x35efc70 (0x7fa19292cc70 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1089205Z frame #10: <unknown function> + 0x35f03e9 (0x7fa19292d3e9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1089862Z frame #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x172 (0x7fa1910f4e62 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1090400Z frame #12: <unknown function> + 0x2ff562 (0x7fa19c01b562 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:37.1090836Z frame #13: <unknown function> + 0x2ff956 (0x7fa19c01b956 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:37.1090964Z frame #14: <unknown function> + 0x1ddc68 (0x55ca7eed5c68 in /opt/conda/bin/python)
2022-11-23T02:55:37.1091144Z frame #15: <unknown function> + 0x199499 (0x55ca7ee91499 in /opt/conda/bin/python)
2022-11-23T02:55:37.1091324Z frame #16: <unknown function> + 0x1995fa (0x55ca7ee915fa in /opt/conda/bin/python)
2022-11-23T02:55:37.1091508Z frame #17: PyNumber_Add + 0x41 (0x55ca7ee3d4b1 in /opt/conda/bin/python)
2022-11-23T02:55:37.1091711Z frame #18: _PyEval_EvalFrameDefault + 0x1008 (0x55ca7eeda098 in /opt/conda/bin/python)
2022-11-23T02:55:37.1091900Z frame #19: <unknown function> + 0x18f742 (0x55ca7ee87742 in /opt/conda/bin/python)
2022-11-23T02:55:37.1092085Z frame #20: _PyObject_Call + 0x20a (0x55ca7ee3ffaa in /opt/conda/bin/python)
2022-11-23T02:55:37.1092291Z frame #21: _PyEval_EvalFrameDefault + 0x26e4 (0x55ca7eedb774 in /opt/conda/bin/python)
2022-11-23T02:55:37.1092456Z frame #22: <unknown function> + 0x18f742 (0x55ca7ee87742 in /opt/conda/bin/python)
2022-11-23T02:55:37.1092689Z frame #23: _PyObject_Call + 0x20a (0x55ca7ee3ffaa in /opt/conda/bin/python)
2022-11-23T02:55:37.1093244Z frame #24: <unknown function> + 0xaa8dba (0x7fa19c7c4dba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:37.1093764Z frame #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7fa19c7c2ffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:37.1094423Z frame #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7fa19c7c62d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:37.1095117Z frame #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7fa19c7c7b16 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:37.1095885Z frame #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7fa193d685cc in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1097106Z frame #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7fa19c7c60c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:37.1097453Z frame #30: <unknown function> + 0x4a24a53 (0x7fa193d61a53 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1098026Z frame #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7fa193d625e8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1098642Z frame #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7fa193d5c8e7 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1099008Z frame #33: <unknown function> + 0x4a545d2 (0x7fa193d915d2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1099599Z frame #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7fa18405890b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:55:37.1099900Z frame #35: <unknown function> + 0xdbbf4 (0x7fa1b3ebebf4 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-11-23T02:55:37.1100219Z frame #36: <unknown function> + 0x76db (0x7fa1d45126db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-11-23T02:55:37.1100498Z frame #37: clone + 0x3f (0x7fa1d423b61f in /lib/x86_64-linux-gnu/libc.so.6)
2022-11-23T02:55:37.1100519Z 
2022-11-23T02:55:37.1100537Z 
2022-11-23T02:55:37.1100666Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:55:37.1112528Z RuntimeError('Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!\nException raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7feb53edb59b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7feb53ed6dfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xced (0x7feb6012213d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7feb6012367f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf2 (0x7feb60124ef2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2e (0x7feb60419b7e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #6: <unknown function> + 0x2a0be3e (0x7feb56d91e3e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #7: <unknown function> + 0x2a0bf46 (0x7feb56d91f46 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7feb60f2bc58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: <unknown function> + 0x35efc70 (0x7feb6279dc70 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: <unknown function> + 0x35f03e9 (0x7feb6279e3e9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x172 (0x7feb60f65e62 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: <unknown function> + 0x2ff562 (0x7feb6be8c562 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #13: <unknown function> + 0x2ff956 (0x7feb6be8c956 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #14: <unknown function> + 0x1ddc68 (0x55564f8f5c68 in /opt/conda/bin/python)\nframe #15: <unknown function> + 0x199499 (0x55564f8b1499 in /opt/conda/bin/python)\nframe #16: <unknown function> + 0x1995fa (0x55564f8b15fa in /opt/conda/bin/python)\nframe #17: PyNumber_Add + 0x41 (0x55564f85d4b1 in /opt/conda/bin/python)\nframe #18: _PyEval_EvalFrameDefault + 0x1008 (0x55564f8fa098 in /opt/conda/bin/python)\nframe #19: <unknown function> + 0x18f742 (0x55564f8a7742 in /opt/conda/bin/python)\nframe #20: _PyObject_Call + 0x20a (0x55564f85ffaa in /opt/conda/bin/python)\nframe #21: _PyEval_EvalFrameDefault + 0x26e4 (0x55564f8fb774 in /opt/conda/bin/python)\nframe #22: <unknown function> + 0x18f742 (0x55564f8a7742 in /opt/conda/bin/python)\nframe #23: _PyObject_Call + 0x20a (0x55564f85ffaa in /opt/conda/bin/python)\nframe #24: <unknown function> + 0xaa8dba (0x7feb6c635dba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7feb6c633ffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7feb6c6372d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7feb6c638b16 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7feb63bd95cc in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7feb6c6370c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #30: <unknown function> + 0x4a24a53 (0x7feb63bd2a53 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7feb63bd35e8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7feb63bcd8e7 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #33: <unknown function> + 0x4a545d2 (0x7feb63c025d2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7feb53ec990b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #35: <unknown function> + 0xdbbf4 (0x7feb83d2fbf4 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #36: <unknown function> + 0x76db (0x7feba43836db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #37: clone + 0x3f (0x7feba40ac61f in /lib/x86_64-linux-gnu/libc.so.6)\n')
2022-11-23T02:55:37.1112825Z Traceback (most recent call last):
2022-11-23T02:55:37.1113177Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:55:37.1113354Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:55:37.1113928Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 5954, in _gpu_add_wrong_gpus
2022-11-23T02:55:37.1114032Z     return x.cpu() + y.cuda()
2022-11-23T02:55:37.1114414Z RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
2022-11-23T02:55:37.1114568Z Exception raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):
2022-11-23T02:55:37.1115128Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7feb53edb59b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:55:37.1115749Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7feb53ed6dfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:55:37.1116249Z frame #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xced (0x7feb6012213d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1116896Z frame #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7feb6012367f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1117650Z frame #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf2 (0x7feb60124ef2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1118164Z frame #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2e (0x7feb60419b7e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1118547Z frame #6: <unknown function> + 0x2a0be3e (0x7feb56d91e3e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:55:37.1118926Z frame #7: <unknown function> + 0x2a0bf46 (0x7feb56d91f46 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:55:37.1119481Z frame #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7feb60f2bc58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1120044Z frame #9: <unknown function> + 0x35efc70 (0x7feb6279dc70 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1120420Z frame #10: <unknown function> + 0x35f03e9 (0x7feb6279e3e9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1120889Z frame #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x172 (0x7feb60f65e62 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1121259Z frame #12: <unknown function> + 0x2ff562 (0x7feb6be8c562 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:37.1121638Z frame #13: <unknown function> + 0x2ff956 (0x7feb6be8c956 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:37.1121831Z frame #14: <unknown function> + 0x1ddc68 (0x55564f8f5c68 in /opt/conda/bin/python)
2022-11-23T02:55:37.1122018Z frame #15: <unknown function> + 0x199499 (0x55564f8b1499 in /opt/conda/bin/python)
2022-11-23T02:55:37.1122192Z frame #16: <unknown function> + 0x1995fa (0x55564f8b15fa in /opt/conda/bin/python)
2022-11-23T02:55:37.1122377Z frame #17: PyNumber_Add + 0x41 (0x55564f85d4b1 in /opt/conda/bin/python)
2022-11-23T02:55:37.1122577Z frame #18: _PyEval_EvalFrameDefault + 0x1008 (0x55564f8fa098 in /opt/conda/bin/python)
2022-11-23T02:55:37.1122741Z frame #19: <unknown function> + 0x18f742 (0x55564f8a7742 in /opt/conda/bin/python)
2022-11-23T02:55:37.1122918Z frame #20: _PyObject_Call + 0x20a (0x55564f85ffaa in /opt/conda/bin/python)
2022-11-23T02:55:37.1123118Z frame #21: _PyEval_EvalFrameDefault + 0x26e4 (0x55564f8fb774 in /opt/conda/bin/python)
2022-11-23T02:55:37.1123295Z frame #22: <unknown function> + 0x18f742 (0x55564f8a7742 in /opt/conda/bin/python)
2022-11-23T02:55:37.1123484Z frame #23: _PyObject_Call + 0x20a (0x55564f85ffaa in /opt/conda/bin/python)
2022-11-23T02:55:37.1123868Z frame #24: <unknown function> + 0xaa8dba (0x7feb6c635dba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:37.1124367Z frame #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7feb6c633ffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:37.1125185Z frame #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7feb6c6372d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:37.1125922Z frame #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7feb6c638b16 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:37.1126695Z frame #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7feb63bd95cc in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1127477Z frame #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7feb6c6370c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:37.1127858Z frame #30: <unknown function> + 0x4a24a53 (0x7feb63bd2a53 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1128648Z frame #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7feb63bd35e8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1129308Z frame #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7feb63bcd8e7 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1129680Z frame #33: <unknown function> + 0x4a545d2 (0x7feb63c025d2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1130083Z frame #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7feb53ec990b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:55:37.1130303Z frame #35: <unknown function> + 0xdbbf4 (0x7feb83d2fbf4 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-11-23T02:55:37.1130792Z frame #36: <unknown function> + 0x76db (0x7feba43836db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-11-23T02:55:37.1131061Z frame #37: clone + 0x3f (0x7feba40ac61f in /lib/x86_64-linux-gnu/libc.so.6)
2022-11-23T02:55:37.1131085Z 
2022-11-23T02:55:37.1131103Z 
2022-11-23T02:55:37.1131237Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:55:37.1143778Z RuntimeError('Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!\nException raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f710244f59b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f710244adfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xced (0x7f710e69613d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7f710e69767f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf2 (0x7f710e698ef2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2e (0x7f710e98db7e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #6: <unknown function> + 0x2a0be3e (0x7f7105305e3e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #7: <unknown function> + 0x2a0bf46 (0x7f7105305f46 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7f710f49fc58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: <unknown function> + 0x35efc70 (0x7f7110d11c70 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: <unknown function> + 0x35f03e9 (0x7f7110d123e9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x172 (0x7f710f4d9e62 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: <unknown function> + 0x2ff562 (0x7f711a400562 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #13: <unknown function> + 0x2ff956 (0x7f711a400956 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #14: <unknown function> + 0x1ddc68 (0x55ec04161c68 in /opt/conda/bin/python)\nframe #15: <unknown function> + 0x199499 (0x55ec0411d499 in /opt/conda/bin/python)\nframe #16: <unknown function> + 0x1995fa (0x55ec0411d5fa in /opt/conda/bin/python)\nframe #17: PyNumber_Add + 0x41 (0x55ec040c94b1 in /opt/conda/bin/python)\nframe #18: _PyEval_EvalFrameDefault + 0x1008 (0x55ec04166098 in /opt/conda/bin/python)\nframe #19: <unknown function> + 0x18f742 (0x55ec04113742 in /opt/conda/bin/python)\nframe #20: _PyObject_Call + 0x20a (0x55ec040cbfaa in /opt/conda/bin/python)\nframe #21: _PyEval_EvalFrameDefault + 0x26e4 (0x55ec04167774 in /opt/conda/bin/python)\nframe #22: <unknown function> + 0x18f742 (0x55ec04113742 in /opt/conda/bin/python)\nframe #23: _PyObject_Call + 0x20a (0x55ec040cbfaa in /opt/conda/bin/python)\nframe #24: <unknown function> + 0xaa8dba (0x7f711aba9dba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f711aba7ffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f711abab2d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7f711abacb16 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7f711214d5cc in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f711abab0c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #30: <unknown function> + 0x4a24a53 (0x7f7112146a53 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f71121475e8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f71121418e7 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #33: <unknown function> + 0x4a545d2 (0x7f71121765d2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f710243d90b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #35: <unknown function> + 0xdbbf4 (0x7f71322a3bf4 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #36: <unknown function> + 0x76db (0x7f71528f76db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #37: clone + 0x3f (0x7f715262061f in /lib/x86_64-linux-gnu/libc.so.6)\n')
2022-11-23T02:55:37.1144024Z Traceback (most recent call last):
2022-11-23T02:55:37.1144364Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:55:37.1144551Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:55:37.1145126Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 5954, in _gpu_add_wrong_gpus
2022-11-23T02:55:37.1145249Z     return x.cpu() + y.cuda()
2022-11-23T02:55:37.1145494Z RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
2022-11-23T02:55:37.1145783Z Exception raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):
2022-11-23T02:55:37.1146396Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f710244f59b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:55:37.1147022Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f710244adfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:55:37.1147515Z frame #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xced (0x7f710e69613d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1148121Z frame #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7f710e69767f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1148685Z frame #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf2 (0x7f710e698ef2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1149174Z frame #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2e (0x7f710e98db7e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1149544Z frame #6: <unknown function> + 0x2a0be3e (0x7f7105305e3e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:55:37.1149908Z frame #7: <unknown function> + 0x2a0bf46 (0x7f7105305f46 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:55:37.1150443Z frame #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7f710f49fc58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1150808Z frame #9: <unknown function> + 0x35efc70 (0x7f7110d11c70 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1151177Z frame #10: <unknown function> + 0x35f03e9 (0x7f7110d123e9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1151822Z frame #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x172 (0x7f710f4d9e62 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1152204Z frame #12: <unknown function> + 0x2ff562 (0x7f711a400562 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:37.1152584Z frame #13: <unknown function> + 0x2ff956 (0x7f711a400956 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:37.1152857Z frame #14: <unknown function> + 0x1ddc68 (0x55ec04161c68 in /opt/conda/bin/python)
2022-11-23T02:55:37.1153095Z frame #15: <unknown function> + 0x199499 (0x55ec0411d499 in /opt/conda/bin/python)
2022-11-23T02:55:37.1153264Z frame #16: <unknown function> + 0x1995fa (0x55ec0411d5fa in /opt/conda/bin/python)
2022-11-23T02:55:37.1153454Z frame #17: PyNumber_Add + 0x41 (0x55ec040c94b1 in /opt/conda/bin/python)
2022-11-23T02:55:37.1153660Z frame #18: _PyEval_EvalFrameDefault + 0x1008 (0x55ec04166098 in /opt/conda/bin/python)
2022-11-23T02:55:37.1153844Z frame #19: <unknown function> + 0x18f742 (0x55ec04113742 in /opt/conda/bin/python)
2022-11-23T02:55:37.1154032Z frame #20: _PyObject_Call + 0x20a (0x55ec040cbfaa in /opt/conda/bin/python)
2022-11-23T02:55:37.1154356Z frame #21: _PyEval_EvalFrameDefault + 0x26e4 (0x55ec04167774 in /opt/conda/bin/python)
2022-11-23T02:55:37.1154488Z frame #22: <unknown function> + 0x18f742 (0x55ec04113742 in /opt/conda/bin/python)
2022-11-23T02:55:37.1154925Z frame #23: _PyObject_Call + 0x20a (0x55ec040cbfaa in /opt/conda/bin/python)
2022-11-23T02:55:37.1155306Z frame #24: <unknown function> + 0xaa8dba (0x7f711aba9dba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:37.1155829Z frame #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f711aba7ffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:37.1156539Z frame #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f711abab2d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:37.1157230Z frame #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7f711abacb16 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:37.1157994Z frame #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7f711214d5cc in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1158775Z frame #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f711abab0c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:37.1159302Z frame #30: <unknown function> + 0x4a24a53 (0x7f7112146a53 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1160115Z frame #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f71121475e8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1160743Z frame #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f71121418e7 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1161122Z frame #33: <unknown function> + 0x4a545d2 (0x7f71121765d2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1161608Z frame #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f710243d90b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:55:37.1161824Z frame #35: <unknown function> + 0xdbbf4 (0x7f71322a3bf4 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-11-23T02:55:37.1162149Z frame #36: <unknown function> + 0x76db (0x7f71528f76db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-11-23T02:55:37.1162415Z frame #37: clone + 0x3f (0x7f715262061f in /lib/x86_64-linux-gnu/libc.so.6)
2022-11-23T02:55:37.1162436Z 
2022-11-23T02:55:37.1162454Z 
2022-11-23T02:55:37.1162647Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:55:37.1174858Z RuntimeError('Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!\nException raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fad5392559b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7fad53920dfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xced (0x7fad5fb6c13d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7fad5fb6d67f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf2 (0x7fad5fb6eef2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2e (0x7fad5fe63b7e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #6: <unknown function> + 0x2a0be3e (0x7fad567dbe3e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #7: <unknown function> + 0x2a0bf46 (0x7fad567dbf46 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7fad60975c58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: <unknown function> + 0x35efc70 (0x7fad621e7c70 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: <unknown function> + 0x35f03e9 (0x7fad621e83e9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x172 (0x7fad609afe62 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: <unknown function> + 0x2ff562 (0x7fad6b8d6562 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #13: <unknown function> + 0x2ff956 (0x7fad6b8d6956 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #14: <unknown function> + 0x1ddc68 (0x564f28ea7c68 in /opt/conda/bin/python)\nframe #15: <unknown function> + 0x199499 (0x564f28e63499 in /opt/conda/bin/python)\nframe #16: <unknown function> + 0x1995fa (0x564f28e635fa in /opt/conda/bin/python)\nframe #17: PyNumber_Add + 0x41 (0x564f28e0f4b1 in /opt/conda/bin/python)\nframe #18: _PyEval_EvalFrameDefault + 0x1008 (0x564f28eac098 in /opt/conda/bin/python)\nframe #19: <unknown function> + 0x18f742 (0x564f28e59742 in /opt/conda/bin/python)\nframe #20: _PyObject_Call + 0x20a (0x564f28e11faa in /opt/conda/bin/python)\nframe #21: _PyEval_EvalFrameDefault + 0x26e4 (0x564f28ead774 in /opt/conda/bin/python)\nframe #22: <unknown function> + 0x18f742 (0x564f28e59742 in /opt/conda/bin/python)\nframe #23: _PyObject_Call + 0x20a (0x564f28e11faa in /opt/conda/bin/python)\nframe #24: <unknown function> + 0xaa8dba (0x7fad6c07fdba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7fad6c07dffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7fad6c0812d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7fad6c082b16 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7fad636235cc in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7fad6c0810c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #30: <unknown function> + 0x4a24a53 (0x7fad6361ca53 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7fad6361d5e8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7fad636178e7 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #33: <unknown function> + 0x4a545d2 (0x7fad6364c5d2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7fad5391390b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #35: <unknown function> + 0xdbbf4 (0x7fad83779bf4 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #36: <unknown function> + 0x76db (0x7fada3dcd6db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #37: clone + 0x3f (0x7fada3af661f in /lib/x86_64-linux-gnu/libc.so.6)\n')
2022-11-23T02:55:37.1175294Z Traceback (most recent call last):
2022-11-23T02:55:37.1175646Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:55:37.1175837Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:55:37.1176231Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 5954, in _gpu_add_wrong_gpus
2022-11-23T02:55:37.1176349Z     return x.cpu() + y.cuda()
2022-11-23T02:55:37.1176585Z RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
2022-11-23T02:55:37.1177035Z Exception raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):
2022-11-23T02:55:37.1177574Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fad5392559b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:55:37.1178173Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7fad53920dfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:55:37.1178652Z frame #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xced (0x7fad5fb6c13d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1179105Z frame #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7fad5fb6d67f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1179899Z frame #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf2 (0x7fad5fb6eef2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1180432Z frame #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2e (0x7fad5fe63b7e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1180824Z frame #6: <unknown function> + 0x2a0be3e (0x7fad567dbe3e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:55:37.1181207Z frame #7: <unknown function> + 0x2a0bf46 (0x7fad567dbf46 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:55:37.1181804Z frame #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7fad60975c58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1182184Z frame #9: <unknown function> + 0x35efc70 (0x7fad621e7c70 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1182560Z frame #10: <unknown function> + 0x35f03e9 (0x7fad621e83e9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1183275Z frame #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x172 (0x7fad609afe62 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1183835Z frame #12: <unknown function> + 0x2ff562 (0x7fad6b8d6562 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:37.1184205Z frame #13: <unknown function> + 0x2ff956 (0x7fad6b8d6956 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:37.1184404Z frame #14: <unknown function> + 0x1ddc68 (0x564f28ea7c68 in /opt/conda/bin/python)
2022-11-23T02:55:37.1184594Z frame #15: <unknown function> + 0x199499 (0x564f28e63499 in /opt/conda/bin/python)
2022-11-23T02:55:37.1184780Z frame #16: <unknown function> + 0x1995fa (0x564f28e635fa in /opt/conda/bin/python)
2022-11-23T02:55:37.1184973Z frame #17: PyNumber_Add + 0x41 (0x564f28e0f4b1 in /opt/conda/bin/python)
2022-11-23T02:55:37.1185187Z frame #18: _PyEval_EvalFrameDefault + 0x1008 (0x564f28eac098 in /opt/conda/bin/python)
2022-11-23T02:55:37.1185371Z frame #19: <unknown function> + 0x18f742 (0x564f28e59742 in /opt/conda/bin/python)
2022-11-23T02:55:37.1185559Z frame #20: _PyObject_Call + 0x20a (0x564f28e11faa in /opt/conda/bin/python)
2022-11-23T02:55:37.1185750Z frame #21: _PyEval_EvalFrameDefault + 0x26e4 (0x564f28ead774 in /opt/conda/bin/python)
2022-11-23T02:55:37.1185932Z frame #22: <unknown function> + 0x18f742 (0x564f28e59742 in /opt/conda/bin/python)
2022-11-23T02:55:37.1186117Z frame #23: _PyObject_Call + 0x20a (0x564f28e11faa in /opt/conda/bin/python)
2022-11-23T02:55:37.1186519Z frame #24: <unknown function> + 0xaa8dba (0x7fad6c07fdba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:37.1187365Z frame #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7fad6c07dffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:37.1188030Z frame #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7fad6c0812d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:37.1188712Z frame #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7fad6c082b16 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:37.1189518Z frame #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7fad636235cc in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1190452Z frame #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7fad6c0810c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:55:37.1190824Z frame #30: <unknown function> + 0x4a24a53 (0x7fad6361ca53 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1191466Z frame #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7fad6361d5e8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1192076Z frame #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7fad636178e7 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1192501Z frame #33: <unknown function> + 0x4a545d2 (0x7fad6364c5d2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:55:37.1192904Z frame #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7fad5391390b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:55:37.1193098Z frame #35: <unknown function> + 0xdbbf4 (0x7fad83779bf4 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-11-23T02:55:37.1193586Z frame #36: <unknown function> + 0x76db (0x7fada3dcd6db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-11-23T02:55:37.1193855Z frame #37: clone + 0x3f (0x7fada3af661f in /lib/x86_64-linux-gnu/libc.so.6)
2022-11-23T02:55:37.1193892Z 
2022-11-23T02:55:37.1193896Z 
2022-11-23T02:55:37.1193992Z ok (8.001s)
2022-11-23T02:55:37.1194011Z 
2022-11-23T02:55:37.1194281Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1194482Z Ran 1 test in 8.001s
2022-11-23T02:55:37.1194482Z 
2022-11-23T02:55:37.1194519Z OK
2022-11-23T02:55:37.1194541Z 
2022-11-23T02:55:37.1194671Z Generating XML reports...
2022-11-23T02:55:37.1195225Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024854.xml
2022-11-23T02:55:37.1195582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1195754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1196128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1196316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1196721Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4emz9xdb
2022-11-23T02:55:37.1196987Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4emz9xdb/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1197008Z 
2022-11-23T02:55:37.1197116Z Running tests...
2022-11-23T02:55:37.1197377Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1197721Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.1198000Z   test_devices_option_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.1198215Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108680
2022-11-23T02:55:37.1198424Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108681
2022-11-23T02:55:37.1198628Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 108682
2022-11-23T02:55:37.1198833Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 108683
2022-11-23T02:55:37.1199241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1199431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1199988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1200185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1200532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1200704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1201080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1201268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1201631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1201804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1202218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1202403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1202901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1203067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1203424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1203610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1203862Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjo4ixyv_
2022-11-23T02:55:37.1204132Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjo4ixyv_/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1204385Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcqzvnhss
2022-11-23T02:55:37.1204639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpujebk88w
2022-11-23T02:55:37.1204905Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcqzvnhss/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1205144Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpujebk88w/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1205383Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1l8_7zfd
2022-11-23T02:55:37.1205633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1l8_7zfd/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1205853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.1206075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.1206295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.1206517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.1206668Z fi_getinfo: -61
2022-11-23T02:55:37.1206783Z fi_getinfo: -61
2022-11-23T02:55:37.1206916Z fi_getinfo: -61
2022-11-23T02:55:37.1207048Z fi_getinfo: -61
2022-11-23T02:55:37.1207143Z ok (4.814s)
2022-11-23T02:55:37.1207162Z 
2022-11-23T02:55:37.1207420Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1207534Z Ran 1 test in 4.815s
2022-11-23T02:55:37.1207552Z 
2022-11-23T02:55:37.1207642Z OK
2022-11-23T02:55:37.1207661Z 
2022-11-23T02:55:37.1207763Z Generating XML reports...
2022-11-23T02:55:37.1208345Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024905.xml
2022-11-23T02:55:37.1208720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1208894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1209259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1209447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1209688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_v1ow490
2022-11-23T02:55:37.1209950Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_v1ow490/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1209969Z 
2022-11-23T02:55:37.1210079Z Running tests...
2022-11-23T02:55:37.1210315Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1210663Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.1210966Z   test_devices_option_mismatch_reverse (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.1211227Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109027
2022-11-23T02:55:37.1211439Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109028
2022-11-23T02:55:37.1211644Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 109029
2022-11-23T02:55:37.1211853Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 109030
2022-11-23T02:55:37.1212219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1212381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1212735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1212918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1213281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1213446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1213978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1214175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1214571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1214775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1215083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1215281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1215646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1215825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1216197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1216382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1216640Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprvmp7zeg
2022-11-23T02:55:37.1217242Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprvmp7zeg/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1217665Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp04o79tjz
2022-11-23T02:55:37.1217982Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp04o79tjz/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1218243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjv1ujq13
2022-11-23T02:55:37.1218508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjv1ujq13/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1218855Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvlcgknbj
2022-11-23T02:55:37.1219017Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvlcgknbj/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1219252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.1219489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.1219716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.1220045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.1220088Z fi_getinfo: -61
2022-11-23T02:55:37.1220223Z fi_getinfo: -61
2022-11-23T02:55:37.1220613Z fi_getinfo: -61
2022-11-23T02:55:37.1220743Z fi_getinfo: -61
2022-11-23T02:55:37.1220839Z ok (4.772s)
2022-11-23T02:55:37.1220858Z 
2022-11-23T02:55:37.1221117Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1221206Z Ran 1 test in 4.772s
2022-11-23T02:55:37.1221242Z 
2022-11-23T02:55:37.1221311Z OK
2022-11-23T02:55:37.1221329Z 
2022-11-23T02:55:37.1221452Z Generating XML reports...
2022-11-23T02:55:37.1221984Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024913.xml
2022-11-23T02:55:37.1222343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1222515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1222888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1223071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1223319Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj3boc9pa
2022-11-23T02:55:37.1223561Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj3boc9pa/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1223602Z 
2022-11-23T02:55:37.1223689Z Running tests...
2022-11-23T02:55:37.1223946Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1224296Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.1224608Z   test_owner_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.1224821Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109374
2022-11-23T02:55:37.1225221Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109375
2022-11-23T02:55:37.1225437Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 109376
2022-11-23T02:55:37.1225642Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 109377
2022-11-23T02:55:37.1225999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1226174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1226552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1226803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1227114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1227340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1227721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1227918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1228262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1228430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1228800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1228991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1229359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1229528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1230047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1230319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1230540Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplpkvvsvj
2022-11-23T02:55:37.1230954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplpkvvsvj/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1231217Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpur53yf75
2022-11-23T02:55:37.1231486Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpur53yf75/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1231735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmporv0zj_c
2022-11-23T02:55:37.1232002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmporv0zj_c/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1232249Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfvc92cpf
2022-11-23T02:55:37.1232517Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfvc92cpf/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1232750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.1232959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.1233186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.1233330Z fi_getinfo: -61
2022-11-23T02:55:37.1233552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.1233648Z ok (10.908s)
2022-11-23T02:55:37.1233821Z 
2022-11-23T02:55:37.1234248Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1234361Z Ran 1 test in 10.908s
2022-11-23T02:55:37.1234380Z 
2022-11-23T02:55:37.1234474Z OK
2022-11-23T02:55:37.1234493Z 
2022-11-23T02:55:37.1234597Z Generating XML reports...
2022-11-23T02:55:37.1235155Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024920.xml
2022-11-23T02:55:37.1235530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1235700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1236080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1236272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1236528Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu51arep5
2022-11-23T02:55:37.1236801Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu51arep5/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1236881Z 
2022-11-23T02:55:37.1236997Z Running tests...
2022-11-23T02:55:37.1237245Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1237607Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.1237924Z   test_owner_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.1238142Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109749
2022-11-23T02:55:37.1238371Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109750
2022-11-23T02:55:37.1238593Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 109751
2022-11-23T02:55:37.1238810Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 109752
2022-11-23T02:55:37.1239183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1239358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1239773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1239965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1240478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1240647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1241009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1241192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1241716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1242077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1242504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1242564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1242924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1243090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1243454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1243641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1243893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl3168ift
2022-11-23T02:55:37.1244160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl3168ift/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1244565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq7vwue63
2022-11-23T02:55:37.1245000Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq7vwue63/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1245236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp56ttfwox
2022-11-23T02:55:37.1245504Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp56ttfwox/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1245754Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp29de7c4y
2022-11-23T02:55:37.1246018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp29de7c4y/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1246249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.1246527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.1246755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.1246967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.1247111Z fi_getinfo: -61
2022-11-23T02:55:37.1247212Z ok (12.855s)
2022-11-23T02:55:37.1247232Z 
2022-11-23T02:55:37.1247496Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1247606Z Ran 1 test in 12.855s
2022-11-23T02:55:37.1247625Z 
2022-11-23T02:55:37.1247719Z OK
2022-11-23T02:55:37.1247738Z 
2022-11-23T02:55:37.1247864Z Generating XML reports...
2022-11-23T02:55:37.1248735Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024934.xml
2022-11-23T02:55:37.1249109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1249270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1249648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1249891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1250154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnpunwg7n
2022-11-23T02:55:37.1250425Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnpunwg7n/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1250445Z 
2022-11-23T02:55:37.1250559Z Running tests...
2022-11-23T02:55:37.1250837Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1251194Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.1251649Z   test_owner_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.1252058Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110125
2022-11-23T02:55:37.1252257Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110126
2022-11-23T02:55:37.1252474Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 110127
2022-11-23T02:55:37.1252688Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 110128
2022-11-23T02:55:37.1253053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1253235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1253619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1253793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1254155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1254320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1254698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1254879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1255240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1255409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1255772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1255956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1256352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1256528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1257127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1257309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1257563Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_3xhp0ih
2022-11-23T02:55:37.1257830Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_3xhp0ih/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1258078Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_qzw8qeg
2022-11-23T02:55:37.1258344Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_qzw8qeg/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1258578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn_qbxhlr
2022-11-23T02:55:37.1258854Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn_qbxhlr/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1259183Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptjy2p7y_
2022-11-23T02:55:37.1259593Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptjy2p7y_/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1259981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.1260207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.1260430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.1260653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.1260810Z fi_getinfo: -61
2022-11-23T02:55:37.1260893Z ok (12.664s)
2022-11-23T02:55:37.1260914Z 
2022-11-23T02:55:37.1261185Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1261293Z Ran 1 test in 12.664s
2022-11-23T02:55:37.1261312Z 
2022-11-23T02:55:37.1261405Z OK
2022-11-23T02:55:37.1261424Z 
2022-11-23T02:55:37.1261541Z Generating XML reports...
2022-11-23T02:55:37.1262144Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024950.xml
2022-11-23T02:55:37.1262520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1262693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1263056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1263241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1263494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmn0lbur_
2022-11-23T02:55:37.1263766Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmn0lbur_/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1263788Z 
2022-11-23T02:55:37.1263897Z Running tests...
2022-11-23T02:55:37.1264289Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1264513Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.1264983Z   test_owner_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.1265194Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110501
2022-11-23T02:55:37.1265388Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110502
2022-11-23T02:55:37.1265592Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 110503
2022-11-23T02:55:37.1265794Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 110504
2022-11-23T02:55:37.1266214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1266491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1266761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1266942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1267295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1267446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1267803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1267984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1268329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1268543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1268895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1269079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1269439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1269599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1269945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1270122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1270374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpso3lnn0x
2022-11-23T02:55:37.1270635Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpso3lnn0x/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1270880Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpry96gxze
2022-11-23T02:55:37.1271138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpry96gxze/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1271378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyjd_jm3_
2022-11-23T02:55:37.1271803Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyjd_jm3_/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1272038Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb7e1p7s7
2022-11-23T02:55:37.1272298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb7e1p7s7/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1272527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.1272753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.1272973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.1273230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.1273459Z fi_getinfo: -61
2022-11-23T02:55:37.1273469Z ok (10.799s)
2022-11-23T02:55:37.1273489Z 
2022-11-23T02:55:37.1273757Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1273852Z Ran 1 test in 10.799s
2022-11-23T02:55:37.1273871Z 
2022-11-23T02:55:37.1273955Z OK
2022-11-23T02:55:37.1273974Z 
2022-11-23T02:55:37.1274094Z Generating XML reports...
2022-11-23T02:55:37.1274778Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025005.xml
2022-11-23T02:55:37.1275177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1275353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1275718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1276024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1276130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7cj863xr
2022-11-23T02:55:37.1276384Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7cj863xr/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1276403Z 
2022-11-23T02:55:37.1276631Z Running tests...
2022-11-23T02:55:37.1276791Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1277104Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.1277405Z   test_rref_as_arg_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.1277657Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110876
2022-11-23T02:55:37.1277868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110877
2022-11-23T02:55:37.1278078Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 110878
2022-11-23T02:55:37.1278267Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 110879
2022-11-23T02:55:37.1278625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1278792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1279148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1279337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1279693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1279863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1280385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1280541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1280912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1281096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1281468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1281706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1282083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1282259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1282618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1282799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1283038Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk26v6k98
2022-11-23T02:55:37.1283448Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk26v6k98/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1283689Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnw_8rztb
2022-11-23T02:55:37.1283942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnw_8rztb/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1284414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphgdprzef
2022-11-23T02:55:37.1284690Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphgdprzef/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1284936Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsspnyp_q
2022-11-23T02:55:37.1285198Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsspnyp_q/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1285408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.1285631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.1285855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.1286077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.1286223Z fi_getinfo: -61
2022-11-23T02:55:37.1286351Z fi_getinfo: -61
2022-11-23T02:55:37.1286487Z fi_getinfo: -61
2022-11-23T02:55:37.1286617Z fi_getinfo: -61
2022-11-23T02:55:37.1286749Z ok (17.863s)
2022-11-23T02:55:37.1286768Z 
2022-11-23T02:55:37.1287184Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1287294Z Ran 1 test in 17.863s
2022-11-23T02:55:37.1287474Z 
2022-11-23T02:55:37.1287559Z OK
2022-11-23T02:55:37.1287578Z 
2022-11-23T02:55:37.1287696Z Generating XML reports...
2022-11-23T02:55:37.1288242Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025019.xml
2022-11-23T02:55:37.1288609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1288785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1289149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1289341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1289594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgwx3nv3l
2022-11-23T02:55:37.1289869Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgwx3nv3l/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1289889Z 
2022-11-23T02:55:37.1289998Z Running tests...
2022-11-23T02:55:37.1290259Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1290765Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.1291061Z   test_rref_as_arg_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.1291373Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111383
2022-11-23T02:55:37.1291590Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111384
2022-11-23T02:55:37.1291683Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 111385
2022-11-23T02:55:37.1291888Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 111386
2022-11-23T02:55:37.1292243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1292411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1292768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1292953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1293292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1293441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1294051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1294228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1294572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1294799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1295111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1295300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1295664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1295834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1296187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1296372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1296796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpki_0kv6r
2022-11-23T02:55:37.1297329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpki_0kv6r/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1297665Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdnq4trh9
2022-11-23T02:55:37.1297797Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdnq4trh9/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1298042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp36w2ouke
2022-11-23T02:55:37.1298293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp36w2ouke/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1298520Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpef5o034l
2022-11-23T02:55:37.1298771Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpef5o034l/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1298996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.1299210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.1299430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.1299644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.1299927Z fi_getinfo: -61
2022-11-23T02:55:37.1300094Z fi_getinfo: -61
2022-11-23T02:55:37.1300213Z fi_getinfo: -61
2022-11-23T02:55:37.1300346Z fi_getinfo: -61
2022-11-23T02:55:37.1300444Z ok (20.536s)
2022-11-23T02:55:37.1300465Z 
2022-11-23T02:55:37.1300728Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1300837Z Ran 1 test in 20.536s
2022-11-23T02:55:37.1300861Z 
2022-11-23T02:55:37.1300951Z OK
2022-11-23T02:55:37.1300970Z 
2022-11-23T02:55:37.1301096Z Generating XML reports...
2022-11-23T02:55:37.1301644Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025040.xml
2022-11-23T02:55:37.1301998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1302175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1302555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1302740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1303143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxah6whbf
2022-11-23T02:55:37.1303478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxah6whbf/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1303500Z 
2022-11-23T02:55:37.1303613Z Running tests...
2022-11-23T02:55:37.1303873Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1304280Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.1304504Z   test_rref_as_arg_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.1305223Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81962 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.725s)
2022-11-23T02:55:37.1305243Z 
2022-11-23T02:55:37.1305492Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1305593Z Ran 1 test in 1.725s
2022-11-23T02:55:37.1305614Z 
2022-11-23T02:55:37.1305716Z OK (skipped=1)
2022-11-23T02:55:37.1305734Z 
2022-11-23T02:55:37.1305915Z Generating XML reports...
2022-11-23T02:55:37.1306436Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025103.xml
2022-11-23T02:55:37.1306790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1306955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1307326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1307493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1307738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppdxshvil
2022-11-23T02:55:37.1307998Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppdxshvil/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1308017Z 
2022-11-23T02:55:37.1308115Z Running tests...
2022-11-23T02:55:37.1308368Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1308710Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.1309011Z   test_rref_as_arg_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.1309224Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111962
2022-11-23T02:55:37.1309417Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111963
2022-11-23T02:55:37.1309622Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 111964
2022-11-23T02:55:37.1309821Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 111965
2022-11-23T02:55:37.1310181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1310347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1310710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1310896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1311238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1311402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1311740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1311917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1312307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1312477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1312832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1313009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1313361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1313525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1313862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1314041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1314462Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptbi003jt
2022-11-23T02:55:37.1314728Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptbi003jt/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1315157Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_jy3joma
2022-11-23T02:55:37.1315287Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_jy3joma/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1315533Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphh1nt51b
2022-11-23T02:55:37.1315793Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphh1nt51b/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1316039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl13_pssb
2022-11-23T02:55:37.1316285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl13_pssb/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1316513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.1316745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.1316967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.1317338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.1317479Z fi_getinfo: -61
2022-11-23T02:55:37.1317611Z fi_getinfo: -61
2022-11-23T02:55:37.1317907Z fi_getinfo: -61
2022-11-23T02:55:37.1318026Z fi_getinfo: -61
2022-11-23T02:55:37.1318121Z ok (20.311s)
2022-11-23T02:55:37.1318140Z 
2022-11-23T02:55:37.1318395Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1318501Z Ran 1 test in 20.311s
2022-11-23T02:55:37.1318520Z 
2022-11-23T02:55:37.1318604Z OK
2022-11-23T02:55:37.1318623Z 
2022-11-23T02:55:37.1318742Z Generating XML reports...
2022-11-23T02:55:37.1319288Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025108.xml
2022-11-23T02:55:37.1319653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1319817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1320194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1320419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1320785Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppurfm5uf
2022-11-23T02:55:37.1321046Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppurfm5uf/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1321065Z 
2022-11-23T02:55:37.1321171Z Running tests...
2022-11-23T02:55:37.1321425Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1321827Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.1322120Z   test_rref_as_arg_synchronization5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.1322331Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112475
2022-11-23T02:55:37.1322536Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112476
2022-11-23T02:55:37.1322741Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 112477
2022-11-23T02:55:37.1322945Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 112478
2022-11-23T02:55:37.1323305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1323473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1323816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1323984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1324380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1324554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1324917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1325100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1325454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1325617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1326157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1326346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1326696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1326869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1327242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1327429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1327687Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn61zmexl
2022-11-23T02:55:37.1327954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn61zmexl/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1328207Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj8ri7b24
2022-11-23T02:55:37.1328473Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj8ri7b24/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1328721Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_cknjxr8
2022-11-23T02:55:37.1329127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_cknjxr8/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1329369Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6d6laycv
2022-11-23T02:55:37.1329617Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6d6laycv/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1329833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.1330044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.1330259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.1330471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.1330653Z fi_getinfo: -61
2022-11-23T02:55:37.1330778Z fi_getinfo: -61
2022-11-23T02:55:37.1330904Z fi_getinfo: -61
2022-11-23T02:55:37.1331036Z fi_getinfo: -61
2022-11-23T02:55:37.1331133Z ok (17.914s)
2022-11-23T02:55:37.1331151Z 
2022-11-23T02:55:37.1331583Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1331688Z Ran 1 test in 17.915s
2022-11-23T02:55:37.1331708Z 
2022-11-23T02:55:37.1331793Z OK
2022-11-23T02:55:37.1331813Z 
2022-11-23T02:55:37.1331935Z Generating XML reports...
2022-11-23T02:55:37.1332467Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025131.xml
2022-11-23T02:55:37.1332839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1333011Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1333395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1333631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1333880Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl41a20zj
2022-11-23T02:55:37.1334300Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl41a20zj/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1334320Z 
2022-11-23T02:55:37.1334423Z Running tests...
2022-11-23T02:55:37.1334664Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1335005Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.1335303Z   test_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.1335511Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112982
2022-11-23T02:55:37.1335843Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112983
2022-11-23T02:55:37.1335930Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 112984
2022-11-23T02:55:37.1336134Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 112985
2022-11-23T02:55:37.1336494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1336996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1337375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1337565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1337923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1338096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1338465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1338657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1339012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1339191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1339547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1339736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1340254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1340418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1340839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1341033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1341279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnzf96571
2022-11-23T02:55:37.1341538Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnzf96571/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1341785Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprmuo1c6p
2022-11-23T02:55:37.1342027Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprmuo1c6p/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1342268Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeuecmo_k
2022-11-23T02:55:37.1342525Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeuecmo_k/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1342883Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsv9k0w09
2022-11-23T02:55:37.1343014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsv9k0w09/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1343295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.1343512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.1343726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.1343851Z fi_getinfo: -61
2022-11-23T02:55:37.1344064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.1344196Z fi_getinfo: -61
2022-11-23T02:55:37.1344323Z fi_getinfo: -61
2022-11-23T02:55:37.1344453Z fi_getinfo: -61
2022-11-23T02:55:37.1344544Z ok (16.778s)
2022-11-23T02:55:37.1344563Z 
2022-11-23T02:55:37.1344818Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1344922Z Ran 1 test in 16.778s
2022-11-23T02:55:37.1344940Z 
2022-11-23T02:55:37.1345014Z OK
2022-11-23T02:55:37.1345032Z 
2022-11-23T02:55:37.1345142Z Generating XML reports...
2022-11-23T02:55:37.1345848Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025152.xml
2022-11-23T02:55:37.1346219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1346393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1346768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1346956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1347210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj20lxp8_
2022-11-23T02:55:37.1347462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj20lxp8_/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1347499Z 
2022-11-23T02:55:37.1347590Z Running tests...
2022-11-23T02:55:37.1347856Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1348206Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.1348522Z   test_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.1348730Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113488
2022-11-23T02:55:37.1348952Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113489
2022-11-23T02:55:37.1349165Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 113490
2022-11-23T02:55:37.1349375Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 113491
2022-11-23T02:55:37.1349778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1349960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1350335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1350519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1350881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1351049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1351562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1351743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1352087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1352282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1352823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1353004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1353364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1353531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1353894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1354075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1354332Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptc2n23lj
2022-11-23T02:55:37.1354584Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptc2n23lj/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1354835Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpza98sceo
2022-11-23T02:55:37.1355101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpza98sceo/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1355659Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphmqv9xf6
2022-11-23T02:55:37.1355920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphmqv9xf6/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1356165Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjbgsm3i7
2022-11-23T02:55:37.1356428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjbgsm3i7/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1356655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.1356879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.1357093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.1357315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.1357458Z fi_getinfo: -61
2022-11-23T02:55:37.1357592Z fi_getinfo: -61
2022-11-23T02:55:37.1357725Z fi_getinfo: -61
2022-11-23T02:55:37.1357857Z fi_getinfo: -61
2022-11-23T02:55:37.1357954Z ok (17.052s)
2022-11-23T02:55:37.1357974Z 
2022-11-23T02:55:37.1358219Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1358330Z Ran 1 test in 17.052s
2022-11-23T02:55:37.1358348Z 
2022-11-23T02:55:37.1358441Z OK
2022-11-23T02:55:37.1358460Z 
2022-11-23T02:55:37.1358581Z Generating XML reports...
2022-11-23T02:55:37.1359178Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025211.xml
2022-11-23T02:55:37.1359555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1359886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1360248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1360414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1360827Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyk9rdvtf
2022-11-23T02:55:37.1361094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyk9rdvtf/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1361115Z 
2022-11-23T02:55:37.1361218Z Running tests...
2022-11-23T02:55:37.1361472Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1361890Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.1362248Z   test_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.1362469Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113997
2022-11-23T02:55:37.1362689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113998
2022-11-23T02:55:37.1362887Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 113999
2022-11-23T02:55:37.1363098Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 114000
2022-11-23T02:55:37.1363475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1363644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1364016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1364203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1364566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1364731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1365250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1365417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1365757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1365916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1366264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1366441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1366817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1366953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1367308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1367551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1367715Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2tmaa3jm
2022-11-23T02:55:37.1367975Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2tmaa3jm/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1368218Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpase_d7__
2022-11-23T02:55:37.1368574Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpase_d7__/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1368823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1e4d60ib
2022-11-23T02:55:37.1369075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1e4d60ib/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1369314Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn4bu8062
2022-11-23T02:55:37.1369568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn4bu8062/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1369771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.1369990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.1370209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.1370431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.1370573Z fi_getinfo: -61
2022-11-23T02:55:37.1370754Z fi_getinfo: -61
2022-11-23T02:55:37.1370881Z fi_getinfo: -61
2022-11-23T02:55:37.1370992Z fi_getinfo: -61
2022-11-23T02:55:37.1371084Z ok (17.046s)
2022-11-23T02:55:37.1371104Z 
2022-11-23T02:55:37.1371355Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1371458Z Ran 1 test in 17.046s
2022-11-23T02:55:37.1371476Z 
2022-11-23T02:55:37.1371560Z OK
2022-11-23T02:55:37.1371579Z 
2022-11-23T02:55:37.1371693Z Generating XML reports...
2022-11-23T02:55:37.1372399Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025231.xml
2022-11-23T02:55:37.1372770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1372928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1373307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1373494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1373823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp02ze9xaa
2022-11-23T02:55:37.1374008Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp02ze9xaa/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1374028Z 
2022-11-23T02:55:37.1374130Z Running tests...
2022-11-23T02:55:37.1374391Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1374745Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.1375200Z   test_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.1375400Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114506
2022-11-23T02:55:37.1375608Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114507
2022-11-23T02:55:37.1375814Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 114508
2022-11-23T02:55:37.1376018Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 114509
2022-11-23T02:55:37.1376378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1376548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1377184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1377372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1377826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1377946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1378321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1378634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1378852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1379014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1379365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1379541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1379890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1380041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1380563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1380805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1381058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0cmf1cf7
2022-11-23T02:55:37.1381316Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0cmf1cf7/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1381582Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr0o0uvvn
2022-11-23T02:55:37.1381874Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr0o0uvvn/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1382122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7z5jdtl4
2022-11-23T02:55:37.1382386Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7z5jdtl4/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1382621Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjxfqo91o
2022-11-23T02:55:37.1382882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjxfqo91o/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1383101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.1383324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.1383691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.1383906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.1384048Z fi_getinfo: -61
2022-11-23T02:55:37.1384177Z fi_getinfo: -61
2022-11-23T02:55:37.1384288Z fi_getinfo: -61
2022-11-23T02:55:37.1384585Z fi_getinfo: -61
2022-11-23T02:55:37.1384679Z ok (16.570s)
2022-11-23T02:55:37.1384699Z 
2022-11-23T02:55:37.1384963Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1385071Z Ran 1 test in 16.571s
2022-11-23T02:55:37.1385093Z 
2022-11-23T02:55:37.1385177Z OK
2022-11-23T02:55:37.1385196Z 
2022-11-23T02:55:37.1385317Z Generating XML reports...
2022-11-23T02:55:37.1385847Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025251.xml
2022-11-23T02:55:37.1386217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1386388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1386759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1386942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1387240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8rk5vdfd
2022-11-23T02:55:37.1387662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8rk5vdfd/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1387849Z 
2022-11-23T02:55:37.1387949Z Running tests...
2022-11-23T02:55:37.1388210Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1388549Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.1388854Z   test_rref_to_here_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.1389065Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115012
2022-11-23T02:55:37.1389274Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115013
2022-11-23T02:55:37.1389483Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 115014
2022-11-23T02:55:37.1389695Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 115015
2022-11-23T02:55:37.1390064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1390283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1390657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1390985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1391332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1391494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1391964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1392025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1392371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1392537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1392887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1393051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1393388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1393546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1393908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1394255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1394508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo9c8fn7q
2022-11-23T02:55:37.1394772Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo9c8fn7q/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1395026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9akj89wt
2022-11-23T02:55:37.1395287Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9akj89wt/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1395521Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnn7g90kw
2022-11-23T02:55:37.1395851Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnn7g90kw/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1396025Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz9i1aw_t
2022-11-23T02:55:37.1396283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz9i1aw_t/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1396554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.1396784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.1397010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.1397375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.1397503Z fi_getinfo: -61
2022-11-23T02:55:37.1397630Z fi_getinfo: -61
2022-11-23T02:55:37.1397755Z fi_getinfo: -61
2022-11-23T02:55:37.1397879Z fi_getinfo: -61
2022-11-23T02:55:37.1397974Z ok (17.928s)
2022-11-23T02:55:37.1397993Z 
2022-11-23T02:55:37.1398247Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1398349Z Ran 1 test in 17.928s
2022-11-23T02:55:37.1398367Z 
2022-11-23T02:55:37.1398437Z OK
2022-11-23T02:55:37.1398466Z 
2022-11-23T02:55:37.1398568Z Generating XML reports...
2022-11-23T02:55:37.1399094Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025310.xml
2022-11-23T02:55:37.1399496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1399662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1400025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1400291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1400617Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd2bcx06v
2022-11-23T02:55:37.1400880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd2bcx06v/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1400900Z 
2022-11-23T02:55:37.1400990Z Running tests...
2022-11-23T02:55:37.1401256Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1401610Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.1401925Z   test_rref_to_here_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.1402142Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115519
2022-11-23T02:55:37.1402357Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115520
2022-11-23T02:55:37.1402570Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 115521
2022-11-23T02:55:37.1402780Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 115522
2022-11-23T02:55:37.1403146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1403459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1403823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1404004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1404352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1404511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1404863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1405042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1405382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1405530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1405926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1406109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1406464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1406625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1406980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1407161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1407410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp53tcmu0x
2022-11-23T02:55:37.1407664Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp53tcmu0x/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1407894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpujeejg8w
2022-11-23T02:55:37.1408148Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpujeejg8w/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1408431Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw8bd6zno
2022-11-23T02:55:37.1408683Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw8bd6zno/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1408918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxfg31vth
2022-11-23T02:55:37.1409168Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxfg31vth/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1409390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.1409604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.1409804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.1410017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.1410157Z fi_getinfo: -61
2022-11-23T02:55:37.1410286Z fi_getinfo: -61
2022-11-23T02:55:37.1410416Z fi_getinfo: -61
2022-11-23T02:55:37.1410543Z fi_getinfo: -61
2022-11-23T02:55:37.1410642Z ok (20.691s)
2022-11-23T02:55:37.1410661Z 
2022-11-23T02:55:37.1410914Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1411006Z Ran 1 test in 20.691s
2022-11-23T02:55:37.1411024Z 
2022-11-23T02:55:37.1411104Z OK
2022-11-23T02:55:37.1411122Z 
2022-11-23T02:55:37.1411236Z Generating XML reports...
2022-11-23T02:55:37.1411764Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025331.xml
2022-11-23T02:55:37.1412121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1412293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1412656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1412845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1413076Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnatmrpp4
2022-11-23T02:55:37.1413339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnatmrpp4/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1413358Z 
2022-11-23T02:55:37.1413460Z Running tests...
2022-11-23T02:55:37.1413713Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1414056Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.1414536Z   test_rref_to_here_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.1414805Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116032
2022-11-23T02:55:37.1415027Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116033
2022-11-23T02:55:37.1415357Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 116034
2022-11-23T02:55:37.1415570Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 116035
2022-11-23T02:55:37.1415811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1415985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1416366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1416549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1417087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1417264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1417873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1418039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1418562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1418733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1419096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1419283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1419645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1419816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1420250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1420471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1420609Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplj5_3zxf
2022-11-23T02:55:37.1420863Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxxduzsjw
2022-11-23T02:55:37.1421279Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplj5_3zxf/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1421536Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxxduzsjw/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1421776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1va1dmbt
2022-11-23T02:55:37.1422033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1va1dmbt/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1422273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy1b6eojf
2022-11-23T02:55:37.1422526Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy1b6eojf/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1422731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.1422948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.1423164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.1423368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.1423501Z fi_getinfo: -61
2022-11-23T02:55:37.1423627Z fi_getinfo: -61
2022-11-23T02:55:37.1423756Z fi_getinfo: -61
2022-11-23T02:55:37.1423882Z fi_getinfo: -61
2022-11-23T02:55:37.1423960Z ok (17.923s)
2022-11-23T02:55:37.1424039Z 
2022-11-23T02:55:37.1424304Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1424417Z Ran 1 test in 17.923s
2022-11-23T02:55:37.1424436Z 
2022-11-23T02:55:37.1424525Z OK
2022-11-23T02:55:37.1424543Z 
2022-11-23T02:55:37.1424658Z Generating XML reports...
2022-11-23T02:55:37.1425178Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025355.xml
2022-11-23T02:55:37.1425709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1425880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1426242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1426433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1426684Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps4v3fnt9
2022-11-23T02:55:37.1427016Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps4v3fnt9/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1427033Z 
2022-11-23T02:55:37.1427124Z Running tests...
2022-11-23T02:55:37.1427382Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1427738Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.1428044Z   test_rref_to_here_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.1428257Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116539
2022-11-23T02:55:37.1428459Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116540
2022-11-23T02:55:37.1428671Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 116541
2022-11-23T02:55:37.1428883Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 116542
2022-11-23T02:55:37.1429255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1429426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1429950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1430130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1430475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1430624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1430975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1431152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1431505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1431861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1432220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1432405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1432759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1432929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1433280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1433521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1433778Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfxb24ybg
2022-11-23T02:55:37.1434046Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfxb24ybg/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1434296Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp23afvnk
2022-11-23T02:55:37.1434709Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp23afvnk/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1434943Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8oxowmk9
2022-11-23T02:55:37.1435193Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8oxowmk9/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1435416Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpooynewan
2022-11-23T02:55:37.1435663Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpooynewan/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1435884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.1436263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.1436474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.1436568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.1436702Z fi_getinfo: -61
2022-11-23T02:55:37.1436829Z fi_getinfo: -61
2022-11-23T02:55:37.1436942Z fi_getinfo: -61
2022-11-23T02:55:37.1437234Z fi_getinfo: -61
2022-11-23T02:55:37.1437329Z ok (20.444s)
2022-11-23T02:55:37.1437348Z 
2022-11-23T02:55:37.1437607Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1437719Z Ran 1 test in 20.444s
2022-11-23T02:55:37.1437733Z 
2022-11-23T02:55:37.1437821Z OK
2022-11-23T02:55:37.1437840Z 
2022-11-23T02:55:37.1437958Z Generating XML reports...
2022-11-23T02:55:37.1438502Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025415.xml
2022-11-23T02:55:37.1438858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1439029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1439402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1439587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1439834Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxo0z2ftk
2022-11-23T02:55:37.1440096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxo0z2ftk/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1440116Z 
2022-11-23T02:55:37.1440216Z Running tests...
2022-11-23T02:55:37.1440472Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1440976Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.1441275Z   test_rref_with_unpickleable_attributes (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.1441486Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117052
2022-11-23T02:55:37.1441694Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117053
2022-11-23T02:55:37.1441900Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 117054
2022-11-23T02:55:37.1442103Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 117055
2022-11-23T02:55:37.1442458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1442669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1443041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1443262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1443561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1443727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1444084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1444265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1444608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1444770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1445126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1445356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1445695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1446032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1446403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1446584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1446835Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzmqpy1we
2022-11-23T02:55:37.1447099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzmqpy1we/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1447353Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5vc3jf74
2022-11-23T02:55:37.1447617Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5vc3jf74/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1447854Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4umblgqu
2022-11-23T02:55:37.1448114Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4umblgqu/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1448360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5v7v5689
2022-11-23T02:55:37.1448616Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5v7v5689/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1448990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.1449210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.1449427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.1449634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.1449771Z fi_getinfo: -61
2022-11-23T02:55:37.1449886Z fi_getinfo: -61
2022-11-23T02:55:37.1450009Z fi_getinfo: -61
2022-11-23T02:55:37.1450135Z fi_getinfo: -61
2022-11-23T02:55:37.1450229Z ok (8.050s)
2022-11-23T02:55:37.1450249Z 
2022-11-23T02:55:37.1450502Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1450601Z Ran 1 test in 8.050s
2022-11-23T02:55:37.1450619Z 
2022-11-23T02:55:37.1450704Z OK
2022-11-23T02:55:37.1450723Z 
2022-11-23T02:55:37.1450826Z Generating XML reports...
2022-11-23T02:55:37.1451350Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025439.xml
2022-11-23T02:55:37.1451700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1452082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1452477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1452663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1452913Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp428zxhq7
2022-11-23T02:55:37.1453176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp428zxhq7/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1453195Z 
2022-11-23T02:55:37.1453298Z Running tests...
2022-11-23T02:55:37.1453545Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1453898Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.1454228Z   test_tensor_view_as_return_value (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.1454421Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117563
2022-11-23T02:55:37.1454686Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117564
2022-11-23T02:55:37.1454895Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 117565
2022-11-23T02:55:37.1455102Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 117566
2022-11-23T02:55:37.1455475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1455784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1456317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1456505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1457129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1457307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1457677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1457860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1458217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1458379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1458728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1458910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1459271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1459442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1459816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1460228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1460403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi2m2spzt
2022-11-23T02:55:37.1460656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi2m2spzt/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1460885Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpewblfdq7
2022-11-23T02:55:37.1461312Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpewblfdq7/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1461559Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt142oy5t
2022-11-23T02:55:37.1461931Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt142oy5t/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1462192Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf2lgy_kr
2022-11-23T02:55:37.1462449Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf2lgy_kr/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1462675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.1462902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.1463121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.1463324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.1463466Z fi_getinfo: -61
2022-11-23T02:55:37.1463599Z fi_getinfo: -61
2022-11-23T02:55:37.1463728Z fi_getinfo: -61
2022-11-23T02:55:37.1463863Z fi_getinfo: -61
2022-11-23T02:55:37.1463961Z ok (9.871s)
2022-11-23T02:55:37.1463981Z 
2022-11-23T02:55:37.1464235Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1464391Z Ran 1 test in 9.872s
2022-11-23T02:55:37.1464411Z 
2022-11-23T02:55:37.1464497Z OK
2022-11-23T02:55:37.1464516Z 
2022-11-23T02:55:37.1464636Z Generating XML reports...
2022-11-23T02:55:37.1465180Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025450.xml
2022-11-23T02:55:37.1465694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1465954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1466222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1466402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1466642Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd2rnp2z8
2022-11-23T02:55:37.1466887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd2rnp2z8/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1466906Z 
2022-11-23T02:55:37.1467137Z Running tests...
2022-11-23T02:55:37.1467258Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1467596Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.1467898Z   test_device_maps_backward_pass (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.1468105Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118690
2022-11-23T02:55:37.1468312Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118691
2022-11-23T02:55:37.1468519Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 118692
2022-11-23T02:55:37.1468708Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 118693
2022-11-23T02:55:37.1469064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1469229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1469587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1469762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1470107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1470265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1470665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1470853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1471188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1471349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1471698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1471871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1472216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1472549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1472909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1473093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1473331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4bn0dc14
2022-11-23T02:55:37.1473641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4bn0dc14/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1473888Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzpxo77ej
2022-11-23T02:55:37.1474183Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzpxo77ej/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1474398Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxrhv3gee
2022-11-23T02:55:37.1474658Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxrhv3gee/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1474899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwe0zko82
2022-11-23T02:55:37.1475158Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwe0zko82/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1475379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.1475742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.1475954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.1476163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.1476299Z fi_getinfo: -61
2022-11-23T02:55:37.1476426Z fi_getinfo: -61
2022-11-23T02:55:37.1476546Z fi_getinfo: -61
2022-11-23T02:55:37.1476668Z fi_getinfo: -61
2022-11-23T02:55:37.1476746Z ok (8.149s)
2022-11-23T02:55:37.1476777Z 
2022-11-23T02:55:37.1477017Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1477116Z Ran 1 test in 8.149s
2022-11-23T02:55:37.1477134Z 
2022-11-23T02:55:37.1477214Z OK
2022-11-23T02:55:37.1477233Z 
2022-11-23T02:55:37.1477348Z Generating XML reports...
2022-11-23T02:55:37.1478009Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221123025502.xml
2022-11-23T02:55:37.1478244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1478405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1478765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1478933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1479172Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvttfhksh
2022-11-23T02:55:37.1479427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvttfhksh/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1479447Z 
2022-11-23T02:55:37.1479592Z Running tests...
2022-11-23T02:55:37.1479853Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1480197Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.1480501Z   test_dist_autograd_sync_streams (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.1480707Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119349
2022-11-23T02:55:37.1481085Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119350
2022-11-23T02:55:37.1481286Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 119351
2022-11-23T02:55:37.1481490Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 119352
2022-11-23T02:55:37.1481898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1482075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1482449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1482686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1483044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1483210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1483561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1483745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1484106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1484272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1484644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1484832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1485183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1485346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1485712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1485882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1486130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjlssmpmc
2022-11-23T02:55:37.1486397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjlssmpmc/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1486644Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdvdp20xd
2022-11-23T02:55:37.1486906Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdvdp20xd/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1487150Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7b4h92xz
2022-11-23T02:55:37.1487404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7b4h92xz/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1487645Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmzw9jksm
2022-11-23T02:55:37.1488202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmzw9jksm/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1488432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.1488652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.1488923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.1489150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.1489287Z fi_getinfo: -61
2022-11-23T02:55:37.1489415Z fi_getinfo: -61
2022-11-23T02:55:37.1489546Z fi_getinfo: -61
2022-11-23T02:55:37.1489662Z fi_getinfo: -61
2022-11-23T02:55:37.1489755Z ok (9.321s)
2022-11-23T02:55:37.1489774Z 
2022-11-23T02:55:37.1490030Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1490136Z Ran 1 test in 9.321s
2022-11-23T02:55:37.1490155Z 
2022-11-23T02:55:37.1490238Z OK
2022-11-23T02:55:37.1490256Z 
2022-11-23T02:55:37.1490370Z Generating XML reports...
2022-11-23T02:55:37.1490932Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221123025513.xml
2022-11-23T02:55:37.1491451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1491606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1492023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1492200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1492439Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiatzql1j
2022-11-23T02:55:37.1492693Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiatzql1j/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1492712Z 
2022-11-23T02:55:37.1492811Z Running tests...
2022-11-23T02:55:37.1493054Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1493390Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:55:37.1493704Z   test_gradients_synchronizations (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:37.1493903Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120008
2022-11-23T02:55:37.1494106Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120009
2022-11-23T02:55:37.1494310Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 120010
2022-11-23T02:55:37.1494677Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 120011
2022-11-23T02:55:37.1495045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1495209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1495581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1495761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1496203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1496277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1496638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1496997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1497360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1497521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1498029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1498201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1498630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:55:37.1498790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:55:37.1499145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:55:37.1499321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:55:37.1499564Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ed8xp9s
2022-11-23T02:55:37.1499817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ed8xp9s/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1500055Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb_d5j6qk
2022-11-23T02:55:37.1500291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeukx790u
2022-11-23T02:55:37.1500647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb_d5j6qk/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1500954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeukx790u/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1501272Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprtxsa2y6
2022-11-23T02:55:37.1501527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprtxsa2y6/_remote_module_non_scriptable.py
2022-11-23T02:55:37.1501750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:55:37.1501971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:55:37.1502187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:55:37.1502400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:55:37.1502538Z fi_getinfo: -61
2022-11-23T02:55:37.1502655Z fi_getinfo: -61
2022-11-23T02:55:37.1502786Z fi_getinfo: -61
2022-11-23T02:55:37.1502915Z fi_getinfo: -61
2022-11-23T02:55:37.1503010Z ok (10.343s)
2022-11-23T02:55:37.1503033Z 
2022-11-23T02:55:37.1503290Z ----------------------------------------------------------------------
2022-11-23T02:55:37.1503396Z Ran 1 test in 10.343s
2022-11-23T02:55:37.1503415Z 
2022-11-23T02:55:37.1503496Z OK
2022-11-23T02:55:37.1503515Z 
2022-11-23T02:55:37.1503630Z Generating XML reports...
2022-11-23T02:55:37.1504317Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221123025525.xml
2022-11-23T02:55:37.1504349Z 
2022-11-23T02:55:37.1504751Z ##[endgroup]
2022-11-23T02:55:37.1505238Z FINISHED PRINTING LOG FILE of distributed/rpc/cuda/test_tensorpipe_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-cuda-test_tensorpipe_agent_oynlluog)
2022-11-23T02:55:37.1505257Z 
2022-11-23T02:55:37.3318864Z 
2022-11-23T02:55:37.3319152Z real	22m39.837s
2022-11-23T02:55:37.3319297Z user	53m29.441s
2022-11-23T02:55:37.3319413Z sys	41m59.397s
2022-11-23T02:55:37.3319565Z + for f in test/distributed/fsdp/*.py
2022-11-23T02:55:37.3319970Z + python test/run_test.py --verbose -i distributed/fsdp/test_checkpoint_wrapper.py
2022-11-23T02:55:39.6874367Z Ignoring disabled issues:  []
2022-11-23T02:55:39.7388662Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T02:55:39.7389242Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T02:55:39.7389654Z Selected tests:
2022-11-23T02:55:39.7389883Z  distributed/fsdp/test_checkpoint_wrapper.py
2022-11-23T02:55:39.7416624Z Prioritized test from test file changes.
2022-11-23T02:55:39.7417483Z reordering tests for PR:
2022-11-23T02:55:39.7417738Z prioritized: []
2022-11-23T02:55:39.7418362Z the rest: ['distributed/fsdp/test_checkpoint_wrapper.py']
2022-11-23T02:55:39.7418592Z 
2022-11-23T02:55:39.7419437Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T02:55:39.7420392Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T02:55:39.7427137Z parallel (file granularity) tests:
2022-11-23T02:55:39.7427435Z  
2022-11-23T02:55:39.7427686Z serial (file granularity) tests:
2022-11-23T02:55:39.7428107Z  distributed/fsdp/test_checkpoint_wrapper.py
2022-11-23T02:55:42.0549655Z Ignoring disabled issues:  []
2022-11-23T02:55:42.0577523Z Ignoring disabled issues:  []
2022-11-23T02:55:42.4417874Z Running distributed/fsdp/test_checkpoint_wrapper.py ... [2022-11-23 02:55:42.441163]
2022-11-23T02:55:42.4419104Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_checkpoint_wrapper.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:55:42.441607]
2022-11-23T02:55:47.3882730Z 
2022-11-23T02:55:47.3883345Z Expand the folded group to see the log file of distributed/fsdp/test_checkpoint_wrapper
2022-11-23T02:55:47.3884944Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_checkpoint_wrapper (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_checkpoint_wrapper_0dazedt2)
2022-11-23T02:55:47.3885315Z 
2022-11-23T02:55:47.3885550Z Running tests...
2022-11-23T02:55:47.3886022Z ----------------------------------------------------------------------
2022-11-23T02:55:47.3886707Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper
2022-11-23T02:55:47.3887088Z   test_apply_activation_checkpointing (__main__.CheckpointWrapperTest)
2022-11-23T02:55:47.3887876Z Ensures that `apply_activation_checkpointing` can be used ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.724s)
2022-11-23T02:55:47.3888862Z   test_checkpoint_wrapper_cpu_offload (__main__.CheckpointWrapperTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:55:47.3889759Z   test_checkpoint_wrapper_kwarg_support (__main__.CheckpointWrapperTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:55:47.3890357Z   test_checkpoint_wrapper_parity (__main__.CheckpointWrapperTest)
2022-11-23T02:55:47.3890776Z Tests that using checkpoint_wrapper or the functional ... ok (0.803s)
2022-11-23T02:55:47.3891504Z   test_forward_missing_attributes (__main__.CheckpointWrapperTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T02:55:47.3892339Z   test_fqn (__main__.CheckpointWrapperTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T02:55:47.3893190Z   test_load_activation_checkpointed_module (__main__.CheckpointWrapperTest) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T02:55:47.3893595Z 
2022-11-23T02:55:47.3893864Z ----------------------------------------------------------------------
2022-11-23T02:55:47.3894193Z Ran 7 tests in 2.536s
2022-11-23T02:55:47.3894355Z 
2022-11-23T02:55:47.3894464Z OK (skipped=6)
2022-11-23T02:55:47.3894603Z 
2022-11-23T02:55:47.3894730Z Generating XML reports...
2022-11-23T02:55:47.3895371Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper/TEST-CheckpointWrapperTest-20221123025544.xml
2022-11-23T02:55:47.3895756Z 
2022-11-23T02:55:47.3898622Z ##[endgroup]
2022-11-23T02:55:47.3899291Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_checkpoint_wrapper (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_checkpoint_wrapper_0dazedt2)
2022-11-23T02:55:47.3899673Z 
2022-11-23T02:55:47.7388351Z 
2022-11-23T02:55:47.7389144Z real	0m10.407s
2022-11-23T02:55:47.7389465Z user	0m17.330s
2022-11-23T02:55:47.7389716Z sys	0m14.617s
2022-11-23T02:55:47.7390024Z + for f in test/distributed/fsdp/*.py
2022-11-23T02:55:47.7390658Z + python test/run_test.py --verbose -i distributed/fsdp/test_distributed_checkpoint.py
2022-11-23T02:55:50.1240260Z Ignoring disabled issues:  []
2022-11-23T02:55:50.1759622Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T02:55:50.1760233Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T02:55:50.1760614Z Selected tests:
2022-11-23T02:55:50.1760920Z  distributed/fsdp/test_distributed_checkpoint.py
2022-11-23T02:55:50.1787498Z Prioritized test from test file changes.
2022-11-23T02:55:50.1787856Z reordering tests for PR:
2022-11-23T02:55:50.1788145Z prioritized: []
2022-11-23T02:55:50.1788701Z the rest: ['distributed/fsdp/test_distributed_checkpoint.py']
2022-11-23T02:55:50.1788961Z 
2022-11-23T02:55:50.1789511Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T02:55:50.1790825Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T02:55:50.1797885Z parallel (file granularity) tests:
2022-11-23T02:55:50.1798174Z  
2022-11-23T02:55:50.1798441Z serial (file granularity) tests:
2022-11-23T02:55:50.1798800Z  distributed/fsdp/test_distributed_checkpoint.py
2022-11-23T02:55:52.4469790Z Ignoring disabled issues:  []
2022-11-23T02:55:52.4564911Z Ignoring disabled issues:  []
2022-11-23T02:55:52.8781604Z Running distributed/fsdp/test_distributed_checkpoint.py ... [2022-11-23 02:55:52.877636]
2022-11-23T02:55:52.8785430Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_distributed_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:55:52.878152]
2022-11-23T02:55:57.0602906Z 
2022-11-23T02:55:57.0604241Z Expand the folded group to see the log file of distributed/fsdp/test_distributed_checkpoint
2022-11-23T02:55:57.0605395Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_distributed_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_distributed_checkpoint_8hsmfg9r)
2022-11-23T02:55:57.0605792Z 
2022-11-23T02:55:57.0605913Z Running tests...
2022-11-23T02:55:57.0606482Z ----------------------------------------------------------------------
2022-11-23T02:55:57.0607067Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint
2022-11-23T02:55:57.0607710Z   test_distributed_checkpoint_state_dict_type_StateDictType_LOCAL_STATE_DICT (__main__.TestDistributedCheckpoint) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:55:57.0608472Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.752s)
2022-11-23T02:55:57.0609506Z   test_distributed_checkpoint_state_dict_type_StateDictType_SHARDED_STATE_DICT (__main__.TestDistributedCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:55:57.0609981Z 
2022-11-23T02:55:57.0610222Z ----------------------------------------------------------------------
2022-11-23T02:55:57.0610557Z Ran 2 tests in 1.754s
2022-11-23T02:55:57.0610723Z 
2022-11-23T02:55:57.0610834Z OK (skipped=2)
2022-11-23T02:55:57.0610991Z 
2022-11-23T02:55:57.0611117Z Generating XML reports...
2022-11-23T02:55:57.0611752Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint/TEST-TestDistributedCheckpoint-20221123025554.xml
2022-11-23T02:55:57.0612142Z 
2022-11-23T02:55:57.0612450Z ##[endgroup]
2022-11-23T02:55:57.0613381Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_distributed_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_distributed_checkpoint_8hsmfg9r)
2022-11-23T02:55:57.0613793Z 
2022-11-23T02:55:57.4118380Z 
2022-11-23T02:55:57.4118829Z real	0m9.673s
2022-11-23T02:55:57.4119060Z user	0m16.564s
2022-11-23T02:55:57.4119408Z sys	0m13.231s
2022-11-23T02:55:57.4119594Z + for f in test/distributed/fsdp/*.py
2022-11-23T02:55:57.4120226Z + python test/run_test.py --verbose -i distributed/fsdp/test_fsdp_apply.py
2022-11-23T02:55:59.7457543Z Ignoring disabled issues:  []
2022-11-23T02:55:59.7974971Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T02:55:59.7975575Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T02:55:59.7975920Z Selected tests:
2022-11-23T02:55:59.7976231Z  distributed/fsdp/test_fsdp_apply.py
2022-11-23T02:55:59.8005345Z Prioritized test from test file changes.
2022-11-23T02:55:59.8005717Z reordering tests for PR:
2022-11-23T02:55:59.8006288Z prioritized: []
2022-11-23T02:55:59.8006813Z the rest: ['distributed/fsdp/test_fsdp_apply.py']
2022-11-23T02:55:59.8007039Z 
2022-11-23T02:55:59.8007590Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T02:55:59.8008529Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T02:55:59.8012817Z parallel (file granularity) tests:
2022-11-23T02:55:59.8013116Z  
2022-11-23T02:55:59.8013382Z serial (file granularity) tests:
2022-11-23T02:55:59.8013683Z  distributed/fsdp/test_fsdp_apply.py
2022-11-23T02:56:02.1455481Z Ignoring disabled issues:  []
2022-11-23T02:56:02.1500087Z Ignoring disabled issues:  []
2022-11-23T02:56:02.6355017Z Running distributed/fsdp/test_fsdp_apply.py ... [2022-11-23 02:56:02.634857]
2022-11-23T02:56:02.6355863Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_apply.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:56:02.635385]
2022-11-23T02:56:06.8217196Z 
2022-11-23T02:56:06.8218283Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_apply
2022-11-23T02:56:06.8219723Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_apply (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_apply_28qvqqxq)
2022-11-23T02:56:06.8220114Z 
2022-11-23T02:56:06.8220252Z Running tests...
2022-11-23T02:56:06.8220840Z ----------------------------------------------------------------------
2022-11-23T02:56:06.8221444Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_apply
2022-11-23T02:56:06.8221870Z   test_apply_in_summon_raises_error (__main__.TestApply)
2022-11-23T02:56:06.8222355Z Tests that calling ``apply()`` on an FSDP instance inside the ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:56:06.8222926Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.748s)
2022-11-23T02:56:06.8223356Z   test_nested_module_apply (__main__.TestApply)
2022-11-23T02:56:06.8224059Z Tests that ``apply()`` modifies parameter values in-place on a ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T02:56:06.8224597Z   test_transformer_module_apply (__main__.TestApply)
2022-11-23T02:56:06.8225318Z Tests that ``apply()`` modifies parameter values in-place on an ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T02:56:06.8225700Z 
2022-11-23T02:56:06.8225951Z ----------------------------------------------------------------------
2022-11-23T02:56:06.8226297Z Ran 3 tests in 1.749s
2022-11-23T02:56:06.8226762Z 
2022-11-23T02:56:06.8226899Z OK (skipped=3)
2022-11-23T02:56:06.8227064Z 
2022-11-23T02:56:06.8227203Z Generating XML reports...
2022-11-23T02:56:06.8227775Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_apply/TEST-TestApply-20221123025604.xml
2022-11-23T02:56:06.8228117Z 
2022-11-23T02:56:06.8228441Z ##[endgroup]
2022-11-23T02:56:06.8229053Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_apply (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_apply_28qvqqxq)
2022-11-23T02:56:06.8229414Z 
2022-11-23T02:56:07.1986807Z 
2022-11-23T02:56:07.1987363Z real	0m9.787s
2022-11-23T02:56:07.1987649Z user	0m17.806s
2022-11-23T02:56:07.1987903Z sys	0m15.890s
2022-11-23T02:56:07.1988196Z + for f in test/distributed/fsdp/*.py
2022-11-23T02:56:07.1988802Z + python test/run_test.py --verbose -i distributed/fsdp/test_fsdp_checkpoint.py
2022-11-23T02:56:09.5931164Z Ignoring disabled issues:  []
2022-11-23T02:56:09.6460377Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T02:56:09.6461477Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T02:56:09.6461785Z Selected tests:
2022-11-23T02:56:09.6462030Z  distributed/fsdp/test_fsdp_checkpoint.py
2022-11-23T02:56:09.6485251Z Prioritized test from test file changes.
2022-11-23T02:56:09.6486507Z reordering tests for PR:
2022-11-23T02:56:09.6487026Z prioritized: []
2022-11-23T02:56:09.6487566Z the rest: ['distributed/fsdp/test_fsdp_checkpoint.py']
2022-11-23T02:56:09.6487800Z 
2022-11-23T02:56:09.6488360Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T02:56:09.6489302Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T02:56:09.6492888Z parallel (file granularity) tests:
2022-11-23T02:56:09.6493175Z  
2022-11-23T02:56:09.6493820Z serial (file granularity) tests:
2022-11-23T02:56:09.6494191Z  distributed/fsdp/test_fsdp_checkpoint.py
2022-11-23T02:56:11.8957925Z Ignoring disabled issues:  []
2022-11-23T02:56:11.9193691Z Ignoring disabled issues:  []
2022-11-23T02:56:12.3945700Z Running distributed/fsdp/test_fsdp_checkpoint.py ... [2022-11-23 02:56:12.393906]
2022-11-23T02:56:12.3947123Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:56:12.394364]
2022-11-23T02:56:16.6369425Z 
2022-11-23T02:56:16.6370204Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_checkpoint
2022-11-23T02:56:16.6371378Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_checkpoint_cirq9cgf)
2022-11-23T02:56:16.6371807Z 
2022-11-23T02:56:16.6371962Z Running tests...
2022-11-23T02:56:16.6372533Z ----------------------------------------------------------------------
2022-11-23T02:56:16.6373024Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint
2022-11-23T02:56:16.6373716Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:56:16.6374458Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.750s)
2022-11-23T02:56:16.6375446Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_True (__main__.TestFSDPCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:56:16.6377285Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_False (__main__.TestFSDPCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:56:16.6378500Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_True (__main__.TestFSDPCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:56:16.6379654Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_False (__main__.TestFSDPCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:56:16.6380753Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_True (__main__.TestFSDPCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:56:16.6382045Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_False (__main__.TestFSDPCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:56:16.6383261Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_True (__main__.TestFSDPCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:56:16.6384448Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_False (__main__.TestFSDPCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:56:16.6385636Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_True (__main__.TestFSDPCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:56:16.6386782Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_False (__main__.TestFSDPCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:56:16.6387995Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_True (__main__.TestFSDPCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:56:16.6389099Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_False (__main__.TestFSDPCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:56:16.6390270Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_True (__main__.TestFSDPCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:56:16.6391423Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_False (__main__.TestFSDPCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:56:16.6392648Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_True (__main__.TestFSDPCheckpoint) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:56:16.6393158Z 
2022-11-23T02:56:16.6393438Z ----------------------------------------------------------------------
2022-11-23T02:56:16.6393763Z Ran 16 tests in 1.782s
2022-11-23T02:56:16.6393935Z 
2022-11-23T02:56:16.6394054Z OK (skipped=16)
2022-11-23T02:56:16.6394220Z 
2022-11-23T02:56:16.6394356Z Generating XML reports...
2022-11-23T02:56:16.6394966Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint/TEST-TestFSDPCheckpoint-20221123025614.xml
2022-11-23T02:56:16.6395347Z 
2022-11-23T02:56:16.6395675Z ##[endgroup]
2022-11-23T02:56:16.6396310Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_checkpoint_cirq9cgf)
2022-11-23T02:56:16.6396690Z 
2022-11-23T02:56:16.9929350Z 
2022-11-23T02:56:16.9930074Z real	0m9.794s
2022-11-23T02:56:16.9930418Z user	0m16.758s
2022-11-23T02:56:16.9930702Z sys	0m13.318s
2022-11-23T02:56:16.9930990Z + for f in test/distributed/fsdp/*.py
2022-11-23T02:56:16.9932088Z + python test/run_test.py --verbose -i distributed/fsdp/test_fsdp_clip_grad_norm.py
2022-11-23T02:56:19.3832310Z Ignoring disabled issues:  []
2022-11-23T02:56:19.4363187Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T02:56:19.4363859Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T02:56:19.4364149Z Selected tests:
2022-11-23T02:56:19.4364440Z  distributed/fsdp/test_fsdp_clip_grad_norm.py
2022-11-23T02:56:19.4392878Z Prioritized test from test file changes.
2022-11-23T02:56:19.4394127Z reordering tests for PR:
2022-11-23T02:56:19.4394505Z prioritized: []
2022-11-23T02:56:19.4395068Z the rest: ['distributed/fsdp/test_fsdp_clip_grad_norm.py']
2022-11-23T02:56:19.4395278Z 
2022-11-23T02:56:19.4395859Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T02:56:19.4396829Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T02:56:19.4402928Z parallel (file granularity) tests:
2022-11-23T02:56:19.4403245Z  
2022-11-23T02:56:19.4403492Z serial (file granularity) tests:
2022-11-23T02:56:19.4403833Z  distributed/fsdp/test_fsdp_clip_grad_norm.py
2022-11-23T02:56:21.6963698Z Ignoring disabled issues:  []
2022-11-23T02:56:21.6985610Z Ignoring disabled issues:  []
2022-11-23T02:56:22.1242233Z Running distributed/fsdp/test_fsdp_clip_grad_norm.py ... [2022-11-23 02:56:22.123641]
2022-11-23T02:56:22.1243378Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_clip_grad_norm.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:56:22.124095]
2022-11-23T02:56:26.2989097Z 
2022-11-23T02:56:26.2990028Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_clip_grad_norm
2022-11-23T02:56:26.2991558Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_clip_grad_norm (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_clip_grad_norm_8wsivp3_)
2022-11-23T02:56:26.2991974Z 
2022-11-23T02:56:26.2992089Z Running tests...
2022-11-23T02:56:26.2992650Z ----------------------------------------------------------------------
2022-11-23T02:56:26.2993214Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm
2022-11-23T02:56:26.2993651Z   test_ddp_parity (__main__.TestClipGradNorm)
2022-11-23T02:56:26.2994123Z Tests FSDP with ``FullyShardedDataParallel.clip_grad_norm_()` against ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:56:26.2994794Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.736s)
2022-11-23T02:56:26.2995513Z   test_non_root (__main__.TestClipGradNorm)
2022-11-23T02:56:26.2996312Z Tests that calling ``clip_grad_norm_()`` on a non-root FSDP instance ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T02:56:26.2996626Z 
2022-11-23T02:56:26.2996895Z ----------------------------------------------------------------------
2022-11-23T02:56:26.2997204Z Ran 2 tests in 1.737s
2022-11-23T02:56:26.2997429Z 
2022-11-23T02:56:26.2997475Z OK (skipped=2)
2022-11-23T02:56:26.2997630Z 
2022-11-23T02:56:26.2997754Z Generating XML reports...
2022-11-23T02:56:26.2998349Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestClipGradNorm-20221123025624.xml
2022-11-23T02:56:26.2998710Z 
2022-11-23T02:56:26.2999027Z ##[endgroup]
2022-11-23T02:56:26.2999662Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_clip_grad_norm (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_clip_grad_norm_8wsivp3_)
2022-11-23T02:56:26.3000029Z 
2022-11-23T02:56:26.6505132Z 
2022-11-23T02:56:26.6506400Z real	0m9.657s
2022-11-23T02:56:26.6506952Z user	0m16.449s
2022-11-23T02:56:26.6507194Z sys	0m14.051s
2022-11-23T02:56:26.6507466Z + for f in test/distributed/fsdp/*.py
2022-11-23T02:56:26.6508104Z + python test/run_test.py --verbose -i distributed/fsdp/test_fsdp_comm.py
2022-11-23T02:56:29.0771593Z Ignoring disabled issues:  []
2022-11-23T02:56:29.1304301Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T02:56:29.1304898Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T02:56:29.1305243Z Selected tests:
2022-11-23T02:56:29.1305532Z  distributed/fsdp/test_fsdp_comm.py
2022-11-23T02:56:29.1329450Z Prioritized test from test file changes.
2022-11-23T02:56:29.1329790Z reordering tests for PR:
2022-11-23T02:56:29.1330080Z prioritized: []
2022-11-23T02:56:29.1330620Z the rest: ['distributed/fsdp/test_fsdp_comm.py']
2022-11-23T02:56:29.1330845Z 
2022-11-23T02:56:29.1331411Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T02:56:29.1332346Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T02:56:29.1338995Z parallel (file granularity) tests:
2022-11-23T02:56:29.1339309Z  
2022-11-23T02:56:29.1339681Z serial (file granularity) tests:
2022-11-23T02:56:29.1339979Z  distributed/fsdp/test_fsdp_comm.py
2022-11-23T02:56:31.4643404Z Ignoring disabled issues:  []
2022-11-23T02:56:31.4684069Z Ignoring disabled issues:  []
2022-11-23T02:56:31.9241545Z Running distributed/fsdp/test_fsdp_comm.py ... [2022-11-23 02:56:31.923394]
2022-11-23T02:56:31.9242462Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:56:31.923841]
2022-11-23T02:56:36.1005582Z 
2022-11-23T02:56:36.1006241Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_comm
2022-11-23T02:56:36.1007339Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_comm_f7eq7wkn)
2022-11-23T02:56:36.1007706Z 
2022-11-23T02:56:36.1007802Z Running tests...
2022-11-23T02:56:36.1008333Z ----------------------------------------------------------------------
2022-11-23T02:56:36.1008901Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_comm
2022-11-23T02:56:36.1009451Z   test_communication_nested_model_False_use_no_sync_False_sharding_strategy_None (__main__.TestCommunication)
2022-11-23T02:56:36.1010425Z Tests FSDP's communication cost in terms of calls to collective ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:56:36.1011125Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.749s)
2022-11-23T02:56:36.1011873Z   test_communication_nested_model_False_use_no_sync_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2022-11-23T02:56:36.1012667Z Tests FSDP's communication cost in terms of calls to collective ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:56:36.1013241Z   test_communication_nested_model_False_use_no_sync_True_sharding_strategy_None (__main__.TestCommunication)
2022-11-23T02:56:36.1013984Z Tests FSDP's communication cost in terms of calls to collective ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:56:36.1014638Z   test_communication_nested_model_False_use_no_sync_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2022-11-23T02:56:36.1015612Z Tests FSDP's communication cost in terms of calls to collective ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:56:36.1016364Z   test_communication_nested_model_True_use_no_sync_False_sharding_strategy_None (__main__.TestCommunication)
2022-11-23T02:56:36.1017585Z Tests FSDP's communication cost in terms of calls to collective ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:56:36.1018393Z   test_communication_nested_model_True_use_no_sync_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2022-11-23T02:56:36.1019167Z Tests FSDP's communication cost in terms of calls to collective ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:56:36.1019762Z   test_communication_nested_model_True_use_no_sync_True_sharding_strategy_None (__main__.TestCommunication)
2022-11-23T02:56:36.1020658Z Tests FSDP's communication cost in terms of calls to collective ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:56:36.1021312Z   test_communication_nested_model_True_use_no_sync_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2022-11-23T02:56:36.1022109Z Tests FSDP's communication cost in terms of calls to collective ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:56:36.1022487Z 
2022-11-23T02:56:36.1022762Z ----------------------------------------------------------------------
2022-11-23T02:56:36.1023076Z Ran 8 tests in 1.766s
2022-11-23T02:56:36.1023248Z 
2022-11-23T02:56:36.1023516Z OK (skipped=8)
2022-11-23T02:56:36.1023670Z 
2022-11-23T02:56:36.1023795Z Generating XML reports...
2022-11-23T02:56:36.1024390Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_comm/TEST-TestCommunication-20221123025633.xml
2022-11-23T02:56:36.1024714Z 
2022-11-23T02:56:36.1025042Z ##[endgroup]
2022-11-23T02:56:36.1025615Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_comm_f7eq7wkn)
2022-11-23T02:56:36.1025958Z 
2022-11-23T02:56:36.4569572Z 
2022-11-23T02:56:36.4569920Z real	0m9.806s
2022-11-23T02:56:36.4570212Z user	0m18.413s
2022-11-23T02:56:36.4570466Z sys	0m15.563s
2022-11-23T02:56:36.4570753Z + for f in test/distributed/fsdp/*.py
2022-11-23T02:56:36.4571368Z + python test/run_test.py --verbose -i distributed/fsdp/test_fsdp_comm_hooks.py
2022-11-23T02:56:38.8474811Z Ignoring disabled issues:  []
2022-11-23T02:56:38.9003458Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T02:56:38.9004411Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T02:56:38.9004819Z Selected tests:
2022-11-23T02:56:38.9005115Z  distributed/fsdp/test_fsdp_comm_hooks.py
2022-11-23T02:56:38.9028629Z Prioritized test from test file changes.
2022-11-23T02:56:38.9028978Z reordering tests for PR:
2022-11-23T02:56:38.9029260Z prioritized: []
2022-11-23T02:56:38.9029785Z the rest: ['distributed/fsdp/test_fsdp_comm_hooks.py']
2022-11-23T02:56:38.9030011Z 
2022-11-23T02:56:38.9030557Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T02:56:38.9031514Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T02:56:38.9038620Z parallel (file granularity) tests:
2022-11-23T02:56:38.9038932Z  
2022-11-23T02:56:38.9039194Z serial (file granularity) tests:
2022-11-23T02:56:38.9039512Z  distributed/fsdp/test_fsdp_comm_hooks.py
2022-11-23T02:56:41.2261731Z Ignoring disabled issues:  []
2022-11-23T02:56:41.2585226Z Ignoring disabled issues:  []
2022-11-23T02:56:41.6179852Z Running distributed/fsdp/test_fsdp_comm_hooks.py ... [2022-11-23 02:56:41.617338]
2022-11-23T02:56:41.6181059Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm_hooks.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:56:41.617893]
2022-11-23T02:56:45.9059666Z 
2022-11-23T02:56:45.9060736Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_comm_hooks
2022-11-23T02:56:45.9062646Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm_hooks (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_comm_hooks_zlzkmzp5)
2022-11-23T02:56:45.9063140Z 
2022-11-23T02:56:45.9063262Z Running tests...
2022-11-23T02:56:45.9063727Z ----------------------------------------------------------------------
2022-11-23T02:56:45.9064284Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks
2022-11-23T02:56:45.9064912Z   test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:56:45.9065643Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.766s)
2022-11-23T02:56:45.9066534Z   test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T02:56:45.9067617Z   test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T02:56:45.9068608Z   test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T02:56:45.9069626Z   test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T02:56:45.9070657Z   test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T02:56:45.9071482Z   test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2022-11-23T02:56:45.9072522Z Tests FSDP's default communication hook's behavior and correctness. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:56:45.9073200Z   test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2022-11-23T02:56:45.9074007Z Tests FSDP's default communication hook's behavior and correctness. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:56:45.9074658Z   test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2022-11-23T02:56:45.9075490Z Tests FSDP's default communication hook's behavior and correctness. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:56:45.9076134Z   test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2022-11-23T02:56:45.9076942Z Tests FSDP's communication hook interface behavior. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:56:45.9077723Z   test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2022-11-23T02:56:45.9078523Z Tests FSDP's communication hook interface behavior. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:56:45.9079168Z   test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2022-11-23T02:56:45.9079966Z Tests FSDP's communication hook interface behavior. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:56:45.9080625Z   test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2022-11-23T02:56:45.9081412Z Tests FSDP's communication hook interface behavior. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:56:45.9082047Z   test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2022-11-23T02:56:45.9082849Z Tests FSDP's communication hook interface behavior. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:56:45.9083500Z   test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2022-11-23T02:56:45.9084299Z Tests FSDP's communication hook interface behavior. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T02:56:45.9085241Z   test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T02:56:45.9086255Z   test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T02:56:45.9087297Z   test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T02:56:45.9088452Z   test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T02:56:45.9089487Z   test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T02:56:45.9090602Z   test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T02:56:45.9091206Z   test_registering_hook_non_root_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2022-11-23T02:56:45.9091993Z Tests FSDP's communication hook registering for submodules. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T02:56:45.9092677Z   test_registering_hook_non_root_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2022-11-23T02:56:45.9093378Z Tests FSDP's communication hook registering for submodules. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T02:56:45.9094056Z   test_registering_hook_non_root_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2022-11-23T02:56:45.9094860Z Tests FSDP's communication hook registering for submodules. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T02:56:45.9095425Z   test_registering_hook_submodules_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2022-11-23T02:56:45.9096194Z Tests FSDP's communication hook registering for submodules. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T02:56:45.9097184Z   test_registering_hook_submodules_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2022-11-23T02:56:45.9098085Z Tests FSDP's communication hook registering for submodules. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T02:56:45.9098688Z   test_registering_hook_submodules_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2022-11-23T02:56:45.9099473Z Tests FSDP's communication hook registering for submodules. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T02:56:45.9099742Z 
2022-11-23T02:56:45.9100007Z ----------------------------------------------------------------------
2022-11-23T02:56:45.9100316Z Ran 27 tests in 1.804s
2022-11-23T02:56:45.9100479Z 
2022-11-23T02:56:45.9100587Z OK (skipped=27)
2022-11-23T02:56:45.9100741Z 
2022-11-23T02:56:45.9100867Z Generating XML reports...
2022-11-23T02:56:45.9101499Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks/TEST-TestCommunicationHooks-20221123025643.xml
2022-11-23T02:56:45.9101875Z 
2022-11-23T02:56:45.9102250Z ##[endgroup]
2022-11-23T02:56:45.9102859Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm_hooks (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_comm_hooks_zlzkmzp5)
2022-11-23T02:56:45.9103220Z 
2022-11-23T02:56:46.2606085Z 
2022-11-23T02:56:46.2606412Z real	0m9.804s
2022-11-23T02:56:46.2606686Z user	0m16.717s
2022-11-23T02:56:46.2606954Z sys	0m13.924s
2022-11-23T02:56:46.2607222Z + for f in test/distributed/fsdp/*.py
2022-11-23T02:56:46.2607729Z + python test/run_test.py --verbose -i distributed/fsdp/test_fsdp_core.py
2022-11-23T02:56:48.6434471Z Ignoring disabled issues:  []
2022-11-23T02:56:48.6962320Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T02:56:48.6963186Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T02:56:48.6963604Z Selected tests:
2022-11-23T02:56:48.6963916Z  distributed/fsdp/test_fsdp_core.py
2022-11-23T02:56:48.6991675Z Prioritized test from test file changes.
2022-11-23T02:56:48.6992005Z reordering tests for PR:
2022-11-23T02:56:48.6992299Z prioritized: []
2022-11-23T02:56:48.6992815Z the rest: ['distributed/fsdp/test_fsdp_core.py']
2022-11-23T02:56:48.6993042Z 
2022-11-23T02:56:48.6993587Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T02:56:48.6994548Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T02:56:48.7000364Z parallel (file granularity) tests:
2022-11-23T02:56:48.7000822Z  
2022-11-23T02:56:48.7001094Z serial (file granularity) tests:
2022-11-23T02:56:48.7001421Z  distributed/fsdp/test_fsdp_core.py
2022-11-23T02:56:50.9844687Z Ignoring disabled issues:  []
2022-11-23T02:56:51.0102922Z Ignoring disabled issues:  []
2022-11-23T02:56:51.4899330Z Running distributed/fsdp/test_fsdp_core.py ... [2022-11-23 02:56:51.489288]
2022-11-23T02:56:51.4900162Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_core.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 02:56:51.489803]
2022-11-23T03:00:27.9421350Z 
2022-11-23T03:00:27.9425941Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_core
2022-11-23T03:00:27.9427059Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_core (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_core_pcw1rry1)
2022-11-23T03:00:27.9427425Z 
2022-11-23T03:00:27.9427565Z Running tests...
2022-11-23T03:00:27.9428106Z ----------------------------------------------------------------------
2022-11-23T03:00:27.9428692Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_core
2022-11-23T03:00:27.9429937Z   test_pre_backward_hook_registration_after_state_dict (__main__.TestHooks)
2022-11-23T03:00:27.9430542Z Tests that FSDP pre-backward hooks are registered on forward pass ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:00:27.9431192Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.705s)
2022-11-23T03:00:27.9431691Z   test_pre_backward_hook_registration_cuda_first_False (__main__.TestHooks)
2022-11-23T03:00:27.9432585Z Tests that FSDP pre-backward hooks are registered on forward pass ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:27.9433156Z   test_pre_backward_hook_registration_cuda_first_True (__main__.TestHooks)
2022-11-23T03:00:27.9433900Z Tests that FSDP pre-backward hooks are registered on forward pass ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:27.9434494Z   test_register_functions_called_cuda_first_False_mixed_precision_False (__main__.TestHooks)
2022-11-23T03:00:27.9435228Z Tests that ``_register_{pre|post}_backward_hooks()`` are called ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:00:27.9435824Z   test_register_functions_called_cuda_first_False_mixed_precision_True (__main__.TestHooks)
2022-11-23T03:00:27.9436550Z Tests that ``_register_{pre|post}_backward_hooks()`` are called ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:27.9437134Z   test_register_functions_called_cuda_first_True_mixed_precision_False (__main__.TestHooks)
2022-11-23T03:00:27.9438182Z Tests that ``_register_{pre|post}_backward_hooks()`` are called ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:00:27.9438802Z   test_register_functions_called_cuda_first_True_mixed_precision_True (__main__.TestHooks)
2022-11-23T03:00:27.9439534Z Tests that ``_register_{pre|post}_backward_hooks()`` are called ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:00:27.9440084Z   test_transformer_no_grad_mixed_precision_False (__main__.TestNoGrad)
2022-11-23T03:00:27.9440799Z Tests that for an FSDP-wrapped transformer model with shared ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:27.9441343Z   test_transformer_no_grad_mixed_precision_True (__main__.TestNoGrad)
2022-11-23T03:00:27.9442073Z Tests that for an FSDP-wrapped transformer model with shared ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:27.9442630Z   test_param_change_after_init_mixed_precision_False (__main__.TestParamInit)
2022-11-23T03:00:27.9443511Z Tests that changing FSDP model parameter values in-place after FSDP ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:27.9444087Z   test_param_change_after_init_mixed_precision_True (__main__.TestParamInit)
2022-11-23T03:00:27.9444848Z Tests that changing FSDP model parameter values in-place after FSDP ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:27.9445404Z   test_delayed_optim_step_offload_false_no_shard (__main__.TestParityWithDDP)
2022-11-23T03:00:27.9446132Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:27.9446922Z   test_delayed_optim_step_offload_false_none (__main__.TestParityWithDDP)
2022-11-23T03:00:27.9447646Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:27.9448208Z   test_delayed_optim_step_offload_false_shard_grad_op (__main__.TestParityWithDDP)
2022-11-23T03:00:27.9448960Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:27.9449530Z   test_delayed_optim_step_offload_true_no_shard (__main__.TestParityWithDDP)
2022-11-23T03:00:27.9450085Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 122853
2022-11-23T03:00:27.9450608Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 122854
2022-11-23T03:00:27.9451075Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 122855
2022-11-23T03:00:27.9451545Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 122856
2022-11-23T03:00:27.9452167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:00:27.9452605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:00:27.9453202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:00:27.9453690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:00:27.9454255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:00:27.9454719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:00:27.9455360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:00:27.9455814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:00:27.9456386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:00:27.9457315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:00:27.9457933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:00:27.9458418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:00:27.9458974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:00:27.9459438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:00:27.9460021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:00:27.9460478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:00:27.9461049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:00:27.9461556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:00:27.9462064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:00:27.9462526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:00:27.9463204Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:00:27.9463900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:00:27.9464600Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:00:27.9465251Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:00:27.9465785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:00:27.9466261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:00:27.9466731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:00:27.9467167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:00:27.9467657Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:27.9468144Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:27.9468622Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:27.9469087Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:27.9470388Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:00:27.9471178Z   warnings.warn(
2022-11-23T03:00:27.9472418Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:00:27.9473219Z   warnings.warn(
2022-11-23T03:00:27.9474381Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:00:27.9475151Z   warnings.warn(
2022-11-23T03:00:27.9476303Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:00:27.9477122Z   warnings.warn(
2022-11-23T03:00:27.9477401Z   File "<string>", line 1, in <module>
2022-11-23T03:00:27.9477704Z   File "<string>", line 1, in <module>
2022-11-23T03:00:27.9478057Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:27.9478434Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:27.9478735Z   File "<string>", line 1, in <module>
2022-11-23T03:00:27.9479020Z   File "<string>", line 1, in <module>
2022-11-23T03:00:27.9479394Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:27.9479777Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:27.9480133Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:27.9480523Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:27.9480903Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:27.9481355Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:27.9481721Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:27.9482089Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:27.9482461Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:27.9482817Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:27.9483189Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:27.9483573Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:27.9483952Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:27.9484509Z     self.run()
2022-11-23T03:00:27.9484851Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:27.9485242Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:27.9485598Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:27.9485940Z     self.run()
2022-11-23T03:00:27.9486290Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:27.9486611Z     self.run()
2022-11-23T03:00:27.9486953Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:27.9487319Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:27.9487688Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:27.9488021Z     self.run()
2022-11-23T03:00:27.9488357Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:27.9488727Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:27.9489079Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:27.9489524Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:27.9490080Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:27.9490450Z     self.run_test(test_name, pipe)
2022-11-23T03:00:27.9490817Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:27.9491192Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:27.9491682Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:27.9492137Z     self.run_test(test_name, pipe)
2022-11-23T03:00:27.9492647Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:27.9493030Z     self.run_test(test_name, pipe)
2022-11-23T03:00:27.9493545Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:27.9493954Z     getattr(self, test_name)()
2022-11-23T03:00:27.9494439Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:27.9494871Z     self.run_test(test_name, pipe)
2022-11-23T03:00:27.9495396Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:27.9495800Z     getattr(self, test_name)()
2022-11-23T03:00:27.9496312Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:27.9496688Z     getattr(self, test_name)()
2022-11-23T03:00:27.9497628Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:27.9497993Z     fn()
2022-11-23T03:00:27.9498469Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:27.9498859Z     fn()
2022-11-23T03:00:27.9499357Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:27.9499756Z     getattr(self, test_name)()
2022-11-23T03:00:27.9500248Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:27.9500617Z     fn()
2022-11-23T03:00:27.9501114Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:27.9501490Z     test(self, **param_kwargs)
2022-11-23T03:00:27.9502013Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:27.9502414Z     test(self, **param_kwargs)
2022-11-23T03:00:27.9502911Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:27.9503271Z     fn()
2022-11-23T03:00:27.9503766Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:27.9504171Z     test(self, **param_kwargs)
2022-11-23T03:00:27.9504666Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:27.9505051Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9505570Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:27.9505971Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9506471Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:27.9506865Z     test(self, **param_kwargs)
2022-11-23T03:00:27.9507377Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:27.9507755Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9508250Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:27.9508646Z     self.run_subtests(
2022-11-23T03:00:27.9509026Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:27.9509410Z     self.run_subtests(
2022-11-23T03:00:27.9509919Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:27.9510385Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9510768Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:27.9511142Z     self.run_subtests(
2022-11-23T03:00:27.9511660Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:27.9512051Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:27.9512613Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:27.9513127Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:27.9513560Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:27.9513910Z     self.run_subtests(
2022-11-23T03:00:27.9514416Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:27.9514846Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:27.9515379Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:27.9515814Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:27.9516366Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:27.9516804Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:27.9517313Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:27.9517747Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:27.9518306Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:27.9518724Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:27.9519270Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:27.9519673Z     output = model(*input)
2022-11-23T03:00:27.9520188Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:27.9520558Z     output = model(*input)
2022-11-23T03:00:27.9521083Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:27.9521507Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:27.9522065Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:27.9522455Z     output = model(*input)
2022-11-23T03:00:27.9522923Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:27.9523303Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:27.9523794Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:27.9524183Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:27.9524730Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:27.9525132Z     output = model(*input)
2022-11-23T03:00:27.9525640Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:27.9526043Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:27.9526584Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:27.9527023Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:27.9527598Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:27.9528053Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:27.9528631Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:27.9528998Z     _lazy_init(state, module)
2022-11-23T03:00:27.9529484Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:27.9529875Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:27.9530404Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:27.9530918Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:27.9531500Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:27.9531958Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:27.9532498Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:27.9532905Z     _lazy_init(state, module)
2022-11-23T03:00:27.9533411Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:27.9533814Z     handle.init_flat_param_attributes()
2022-11-23T03:00:27.9534332Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:27.9534740Z     _lazy_init(state, module)
2022-11-23T03:00:27.9535248Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:27.9535623Z     _lazy_init(state, module)
2022-11-23T03:00:27.9536122Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:27.9536522Z     handle.init_flat_param_attributes()
2022-11-23T03:00:27.9537324Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:27.9537712Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9538207Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:27.9538591Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9539080Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:27.9539480Z     handle.init_flat_param_attributes()
2022-11-23T03:00:27.9540004Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:27.9540376Z     handle.init_flat_param_attributes()
2022-11-23T03:00:27.9540893Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:27.9541270Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9541806Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:27.9542170Z     p_assert(
2022-11-23T03:00:27.9542697Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:27.9543057Z     p_assert(
2022-11-23T03:00:27.9543584Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:27.9543986Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9544531Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:27.9544905Z     p_assert(
2022-11-23T03:00:27.9545356Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:27.9545735Z     traceback.print_stack()
2022-11-23T03:00:27.9546232Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:27.9546588Z     traceback.print_stack()
2022-11-23T03:00:27.9547125Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:27.9547477Z     p_assert(
2022-11-23T03:00:27.9547954Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:27.9548323Z     traceback.print_stack()
2022-11-23T03:00:27.9548910Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:27.9549293Z     traceback.print_stack()
2022-11-23T03:00:27.9549653Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:27.9550158Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:27.9550646Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:27.9551134Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:27.9551499Z   File "<string>", line 1, in <module>
2022-11-23T03:00:27.9551870Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:27.9552253Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:27.9552619Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:27.9552999Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:27.9553372Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:27.9553669Z     self.run()
2022-11-23T03:00:27.9553999Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:27.9554359Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:27.9554870Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:27.9555243Z     self.run_test(test_name, pipe)
2022-11-23T03:00:27.9555763Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:27.9556152Z     getattr(self, test_name)()
2022-11-23T03:00:27.9556652Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:27.9557017Z     fn()
2022-11-23T03:00:27.9557500Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:27.9557876Z     test(self, **param_kwargs)
2022-11-23T03:00:27.9558385Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:27.9558771Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9559172Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:27.9559526Z     self.run_subtests(
2022-11-23T03:00:27.9559794Z   File "<string>", line 1, in <module>
2022-11-23T03:00:27.9560312Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:27.9560716Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:27.9561324Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:27.9561756Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:27.9562138Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:27.9562490Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:27.9563038Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:27.9563432Z     output = model(*input)
2022-11-23T03:00:27.9563765Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:27.9564133Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:27.9564638Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:27.9565024Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:27.9565390Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:27.9565724Z     self.run()
2022-11-23T03:00:27.9566303Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:27.9566740Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:27.9567134Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:27.9567498Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:27.9568016Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:27.9568403Z     _lazy_init(state, module)
2022-11-23T03:00:27.9568887Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:27.9569268Z     self.run_test(test_name, pipe)
2022-11-23T03:00:27.9569766Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:27.9570167Z     handle.init_flat_param_attributes()
2022-11-23T03:00:27.9570706Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:27.9571075Z     getattr(self, test_name)()
2022-11-23T03:00:27.9571561Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:27.9571940Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9572449Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:27.9572795Z     fn()
2022-11-23T03:00:27.9573291Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:27.9573668Z     p_assert(
2022-11-23T03:00:27.9574152Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:27.9574543Z     test(self, **param_kwargs)
2022-11-23T03:00:27.9575034Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:27.9575390Z     traceback.print_stack()
2022-11-23T03:00:27.9575912Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:27.9576295Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9576867Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:27.9577240Z     self.run_subtests(
2022-11-23T03:00:27.9577742Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:27.9578162Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:27.9578773Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:27.9579205Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:27.9579763Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:27.9580156Z     output = model(*input)
2022-11-23T03:00:27.9580611Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:27.9580993Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:27.9581536Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:27.9581970Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:27.9582532Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:27.9582923Z     _lazy_init(state, module)
2022-11-23T03:00:27.9583431Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:27.9583956Z     handle.init_flat_param_attributes()
2022-11-23T03:00:27.9584471Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:27.9584849Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9585363Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:27.9585736Z     p_assert(
2022-11-23T03:00:27.9586196Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:27.9586572Z     traceback.print_stack()
2022-11-23T03:00:27.9586838Z   File "<string>", line 1, in <module>
2022-11-23T03:00:27.9587203Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:27.9587574Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:27.9587925Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:27.9588296Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:27.9588680Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:27.9588991Z     self.run()
2022-11-23T03:00:27.9589320Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:27.9589689Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:27.9590195Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:27.9590563Z     self.run_test(test_name, pipe)
2022-11-23T03:00:27.9591081Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:27.9591467Z     getattr(self, test_name)()
2022-11-23T03:00:27.9592010Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:27.9592382Z     fn()
2022-11-23T03:00:27.9592879Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:27.9593258Z     test(self, **param_kwargs)
2022-11-23T03:00:27.9593766Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:27.9594154Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9594553Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:27.9594905Z     self.run_subtests(
2022-11-23T03:00:27.9595399Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:27.9595813Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:27.9596395Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:27.9596822Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:27.9597379Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:27.9597774Z     output = model(*input)
2022-11-23T03:00:27.9598234Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:27.9598618Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:27.9599165Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:27.9599593Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:27.9600152Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:27.9600541Z     _lazy_init(state, module)
2022-11-23T03:00:27.9601041Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:27.9601487Z     handle.init_flat_param_attributes()
2022-11-23T03:00:27.9602001Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:27.9602377Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9602893Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:27.9603270Z     p_assert(
2022-11-23T03:00:27.9603738Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:27.9604116Z     traceback.print_stack()
2022-11-23T03:00:27.9604383Z   File "<string>", line 1, in <module>
2022-11-23T03:00:27.9604751Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:27.9605117Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:27.9605468Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:27.9605837Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:27.9606221Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:27.9606536Z     self.run()
2022-11-23T03:00:27.9606864Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:27.9607224Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:27.9607718Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:27.9608104Z     self.run_test(test_name, pipe)
2022-11-23T03:00:27.9608621Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:27.9609006Z     getattr(self, test_name)()
2022-11-23T03:00:27.9609500Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:27.9609868Z     fn()
2022-11-23T03:00:27.9610416Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:27.9610790Z     test(self, **param_kwargs)
2022-11-23T03:00:27.9611306Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:27.9611691Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9612092Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:27.9614572Z     self.run_subtests(
2022-11-23T03:00:27.9615095Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:27.9615528Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:27.9616131Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:27.9616575Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:27.9617344Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:27.9617750Z     output = model(*input)
2022-11-23T03:00:27.9618215Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:27.9618618Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:27.9619168Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:27.9619607Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:27.9620181Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:27.9620581Z     _lazy_init(state, module)
2022-11-23T03:00:27.9621092Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:27.9621577Z     handle.init_flat_param_attributes()
2022-11-23T03:00:27.9622101Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:27.9622494Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9623012Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:27.9623402Z     p_assert(
2022-11-23T03:00:27.9623874Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:27.9624264Z     traceback.print_stack()
2022-11-23T03:00:27.9624643Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:27.9625141Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:27.9625629Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:27.9626099Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:27.9626478Z   File "<string>", line 1, in <module>
2022-11-23T03:00:27.9626858Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:27.9627213Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:27.9627595Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:27.9627960Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:27.9628351Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:27.9628672Z     self.run()
2022-11-23T03:00:27.9629014Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:27.9629385Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:27.9629882Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:27.9630269Z     self.run_test(test_name, pipe)
2022-11-23T03:00:27.9630795Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:27.9631196Z     getattr(self, test_name)()
2022-11-23T03:00:27.9631770Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:27.9632152Z     fn()
2022-11-23T03:00:27.9632644Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:27.9633028Z     test(self, **param_kwargs)
2022-11-23T03:00:27.9633544Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:27.9633944Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9634405Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:27.9634794Z     self.run_subtests(
2022-11-23T03:00:27.9635308Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:27.9635737Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:27.9636270Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:27.9636686Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:27.9637235Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:27.9637606Z     output = model(*input)
2022-11-23T03:00:27.9638080Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:27.9638464Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:27.9639012Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:27.9639511Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:27.9639829Z   File "<string>", line 1, in <module>
2022-11-23T03:00:27.9640357Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:27.9640724Z     _lazy_init(state, module)
2022-11-23T03:00:27.9641223Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:27.9641621Z     handle.init_flat_param_attributes()
2022-11-23T03:00:27.9641999Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:27.9642359Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:27.9642858Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:27.9643235Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9643574Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:27.9643940Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:27.9644497Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:27.9644859Z     p_assert(
2022-11-23T03:00:27.9645199Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:27.9645532Z     self.run()
2022-11-23T03:00:27.9645996Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:27.9646359Z     traceback.print_stack()
2022-11-23T03:00:27.9646714Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:27.9647088Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:27.9647576Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:27.9647964Z     self.run_test(test_name, pipe)
2022-11-23T03:00:27.9648481Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:27.9648865Z     getattr(self, test_name)()
2022-11-23T03:00:27.9649373Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:27.9649723Z     fn()
2022-11-23T03:00:27.9650206Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:27.9650579Z     test(self, **param_kwargs)
2022-11-23T03:00:27.9651082Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:27.9651523Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9651915Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:27.9652297Z     self.run_subtests(
2022-11-23T03:00:27.9652808Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:27.9653239Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:27.9653775Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:27.9654206Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:27.9654765Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:27.9655138Z     output = model(*input)
2022-11-23T03:00:27.9655412Z   File "<string>", line 1, in <module>
2022-11-23T03:00:27.9655902Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:27.9656289Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:27.9657069Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:27.9657530Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:27.9657940Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:27.9658298Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:27.9658852Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:27.9659256Z     _lazy_init(state, module)
2022-11-23T03:00:27.9659615Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:27.9659970Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:27.9660504Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:27.9660915Z     handle.init_flat_param_attributes()
2022-11-23T03:00:27.9661291Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:27.9661637Z     self.run()
2022-11-23T03:00:27.9662115Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:27.9662483Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9662847Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:27.9663231Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:27.9663788Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:27.9664157Z     p_assert(
2022-11-23T03:00:27.9664622Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:27.9665006Z     self.run_test(test_name, pipe)
2022-11-23T03:00:27.9665485Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:27.9665873Z     traceback.print_stack()
2022-11-23T03:00:27.9666382Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:27.9666754Z     getattr(self, test_name)()
2022-11-23T03:00:27.9667258Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:27.9667619Z     fn()
2022-11-23T03:00:27.9668103Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:27.9668475Z     test(self, **param_kwargs)
2022-11-23T03:00:27.9668753Z   File "<string>", line 1, in <module>
2022-11-23T03:00:27.9669348Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:27.9669730Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9670134Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:27.9670506Z     self.run_subtests(
2022-11-23T03:00:27.9670860Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:27.9671210Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:27.9671738Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:27.9672160Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:27.9672526Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:27.9672896Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:27.9673441Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:27.9673844Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:27.9674322Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:27.9674655Z     self.run()
2022-11-23T03:00:27.9675169Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:27.9675546Z     output = model(*input)
2022-11-23T03:00:27.9675897Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:27.9676264Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:27.9676741Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:27.9677126Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:27.9677632Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:27.9678021Z     self.run_test(test_name, pipe)
2022-11-23T03:00:27.9678547Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:27.9679003Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:27.9679562Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:27.9679932Z     getattr(self, test_name)()
2022-11-23T03:00:27.9680451Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:27.9680837Z     _lazy_init(state, module)
2022-11-23T03:00:27.9681346Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:27.9681689Z     fn()
2022-11-23T03:00:27.9682159Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:27.9682568Z     handle.init_flat_param_attributes()
2022-11-23T03:00:27.9683092Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:27.9683486Z     test(self, **param_kwargs)
2022-11-23T03:00:27.9683982Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:27.9684343Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9684862Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:27.9685259Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9685799Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:27.9686165Z     p_assert(
2022-11-23T03:00:27.9686556Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:27.9686990Z     self.run_subtests(
2022-11-23T03:00:27.9687468Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:27.9687856Z     traceback.print_stack()
2022-11-23T03:00:27.9688377Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:27.9688803Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:27.9689338Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:27.9689766Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:27.9690325Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:27.9690718Z     output = model(*input)
2022-11-23T03:00:27.9691199Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:27.9691590Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:27.9692267Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:27.9692700Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:27.9693277Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:27.9693678Z     _lazy_init(state, module)
2022-11-23T03:00:27.9694166Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:27.9694581Z     handle.init_flat_param_attributes()
2022-11-23T03:00:27.9695098Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:27.9695487Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9696011Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:27.9696401Z     p_assert(
2022-11-23T03:00:27.9697082Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:27.9697461Z     traceback.print_stack()
2022-11-23T03:00:27.9697864Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:27.9698358Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:27.9698827Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:27.9699315Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:27.9699703Z   File "<string>", line 1, in <module>
2022-11-23T03:00:27.9700076Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:27.9700440Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:27.9700829Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:27.9701209Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:27.9701585Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:27.9701927Z     self.run()
2022-11-23T03:00:27.9702264Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:27.9702615Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:27.9703146Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:27.9703539Z     self.run_test(test_name, pipe)
2022-11-23T03:00:27.9704068Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:27.9704442Z     getattr(self, test_name)()
2022-11-23T03:00:27.9705053Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:27.9705440Z     fn()
2022-11-23T03:00:27.9705919Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:27.9706322Z     test(self, **param_kwargs)
2022-11-23T03:00:27.9706838Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:27.9707235Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9707622Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:27.9708006Z     self.run_subtests(
2022-11-23T03:00:27.9708508Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:27.9708914Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:27.9709476Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:27.9709987Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:27.9710616Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:27.9711003Z     output = model(*input)
2022-11-23T03:00:27.9711487Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:27.9711885Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:27.9712415Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:27.9712877Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:27.9713445Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:27.9713842Z     _lazy_init(state, module)
2022-11-23T03:00:27.9714331Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:27.9714748Z     handle.init_flat_param_attributes()
2022-11-23T03:00:27.9715272Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:27.9715639Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9716175Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:27.9716565Z     p_assert(
2022-11-23T03:00:27.9717018Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:27.9717406Z     traceback.print_stack()
2022-11-23T03:00:27.9717699Z   File "<string>", line 1, in <module>
2022-11-23T03:00:27.9718076Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:27.9718433Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:27.9718810Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:27.9719194Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:27.9719567Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:27.9719908Z     self.run()
2022-11-23T03:00:27.9720248Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:27.9720595Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:27.9721121Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:27.9721515Z     self.run_test(test_name, pipe)
2022-11-23T03:00:27.9722044Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:27.9722419Z     getattr(self, test_name)()
2022-11-23T03:00:27.9723002Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:27.9723384Z     fn()
2022-11-23T03:00:27.9723864Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:27.9724263Z     test(self, **param_kwargs)
2022-11-23T03:00:27.9724782Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:27.9725172Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9725556Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:27.9725934Z     self.run_subtests(
2022-11-23T03:00:27.9726434Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:27.9726836Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:27.9727394Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:27.9727881Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:27.9728447Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:27.9728829Z     output = model(*input)
2022-11-23T03:00:27.9729311Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:27.9729702Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:27.9730229Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:27.9730693Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:27.9731263Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:27.9731664Z     _lazy_init(state, module)
2022-11-23T03:00:27.9732158Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:27.9732582Z     handle.init_flat_param_attributes()
2022-11-23T03:00:27.9733106Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:27.9733489Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9734020Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:27.9734418Z     p_assert(
2022-11-23T03:00:27.9734869Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:27.9735254Z     traceback.print_stack()
2022-11-23T03:00:27.9735547Z   File "<string>", line 1, in <module>
2022-11-23T03:00:27.9735841Z   File "<string>", line 1, in <module>
2022-11-23T03:00:27.9736202Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:27.9736584Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:27.9737214Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:27.9737627Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:27.9738091Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:27.9738544Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:27.9738943Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:27.9768285Z     self.run()
2022-11-23T03:00:27.9768694Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:27.9769095Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:27.9769482Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:27.9769835Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:27.9770509Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:27.9770888Z     self.run()
2022-11-23T03:00:27.9771410Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:27.9771786Z     self.run_test(test_name, pipe)
2022-11-23T03:00:27.9772158Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:27.9772531Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:27.9773057Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:27.9773460Z     getattr(self, test_name)()
2022-11-23T03:00:27.9773959Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:27.9774328Z     self.run_test(test_name, pipe)
2022-11-23T03:00:27.9774865Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:27.9775248Z     fn()
2022-11-23T03:00:27.9775831Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:27.9776207Z     getattr(self, test_name)()
2022-11-23T03:00:27.9777028Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:27.9777445Z     test(self, **param_kwargs)
2022-11-23T03:00:27.9777953Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:27.9778322Z     fn()
2022-11-23T03:00:27.9778805Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:27.9779203Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9779599Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:27.9779986Z     self.run_subtests(
2022-11-23T03:00:27.9780516Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:27.9780894Z     test(self, **param_kwargs)
2022-11-23T03:00:27.9781475Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:27.9782046Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:27.9782558Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:27.9782941Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9783442Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:27.9783823Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:27.9784208Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:27.9784556Z     self.run_subtests(
2022-11-23T03:00:27.9785074Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:27.9785440Z     output = model(*input)
2022-11-23T03:00:27.9785935Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:27.9786330Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:27.9786813Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:27.9787169Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:27.9787668Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:27.9788082Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:27.9788690Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:27.9789157Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:27.9789938Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:27.9790347Z     output = model(*input)
2022-11-23T03:00:27.9790856Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:27.9791254Z     _lazy_init(state, module)
2022-11-23T03:00:27.9791736Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:27.9792156Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:27.9792692Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:27.9793112Z     handle.init_flat_param_attributes()
2022-11-23T03:00:27.9793672Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:27.9794189Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:27.9794743Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:27.9795133Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9795963Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:27.9796365Z     _lazy_init(state, module)
2022-11-23T03:00:27.9796903Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:27.9797295Z     p_assert(
2022-11-23T03:00:27.9797763Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:27.9798180Z     handle.init_flat_param_attributes()
2022-11-23T03:00:27.9798692Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:27.9799057Z     traceback.print_stack()
2022-11-23T03:00:27.9799549Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:27.9799933Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9800473Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:27.9800842Z     p_assert(
2022-11-23T03:00:27.9801317Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:27.9801705Z     traceback.print_stack()
2022-11-23T03:00:27.9802083Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:27.9802581Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:27.9803227Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:27.9803676Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:27.9804052Z   File "<string>", line 1, in <module>
2022-11-23T03:00:27.9804339Z   File "<string>", line 1, in <module>
2022-11-23T03:00:27.9804704Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:27.9805048Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:27.9805412Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:27.9805774Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:27.9806127Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:27.9806490Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:27.9806923Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:27.9807412Z     self.run()
2022-11-23T03:00:27.9807751Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:27.9808128Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:27.9808512Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:27.9808862Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:27.9809254Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:27.9809593Z     self.run()
2022-11-23T03:00:27.9810066Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:27.9810669Z     self.run_test(test_name, pipe)
2022-11-23T03:00:27.9811027Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:27.9811372Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:27.9811900Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:27.9812351Z     getattr(self, test_name)()
2022-11-23T03:00:27.9812831Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:27.9813188Z     self.run_test(test_name, pipe)
2022-11-23T03:00:27.9813704Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:27.9814062Z     fn()
2022-11-23T03:00:27.9814513Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:27.9814894Z     getattr(self, test_name)()
2022-11-23T03:00:27.9815583Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:27.9815979Z     test(self, **param_kwargs)
2022-11-23T03:00:27.9816478Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:27.9817068Z     fn()
2022-11-23T03:00:27.9817563Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:27.9817941Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9818616Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:27.9819002Z     test(self, **param_kwargs)
2022-11-23T03:00:27.9819376Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:27.9819747Z     self.run_subtests(
2022-11-23T03:00:27.9820245Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:27.9820628Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9821106Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:27.9821531Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:27.9821963Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:27.9822482Z     self.run_subtests(
2022-11-23T03:00:27.9822997Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:27.9823420Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:27.9823953Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:27.9824358Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:27.9824922Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:27.9825400Z     output = model(*input)
2022-11-23T03:00:27.9826074Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:27.9826491Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:27.9826988Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:27.9827370Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:27.9827885Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:27.9828449Z     output = model(*input)
2022-11-23T03:00:27.9828985Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:27.9829426Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:27.9829958Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:27.9830352Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:27.9830982Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:27.9831515Z     _lazy_init(state, module)
2022-11-23T03:00:27.9832025Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:27.9832652Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:27.9833188Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:27.9833604Z     handle.init_flat_param_attributes()
2022-11-23T03:00:27.9834147Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:27.9834550Z     _lazy_init(state, module)
2022-11-23T03:00:27.9835037Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:27.9835429Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9835944Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:27.9836332Z     handle.init_flat_param_attributes()
2022-11-23T03:00:27.9836888Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:27.9837277Z     p_assert(
2022-11-23T03:00:27.9837538Z   File "<string>", line 1, in <module>
2022-11-23T03:00:27.9838028Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:27.9838415Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9838916Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:27.9839289Z     traceback.print_stack()
2022-11-23T03:00:27.9839835Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:27.9840220Z     p_assert(
2022-11-23T03:00:27.9840544Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:27.9840923Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:27.9841231Z   File "<string>", line 1, in <module>
2022-11-23T03:00:27.9841738Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:27.9842101Z     traceback.print_stack()
2022-11-23T03:00:27.9842469Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:27.9842852Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:27.9843227Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:27.9843603Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:27.9844063Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:27.9844398Z     self.run()
2022-11-23T03:00:27.9844736Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:27.9845115Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:27.9845496Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:27.9845852Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:27.9846241Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:27.9846579Z     self.run()
2022-11-23T03:00:27.9847046Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:27.9847441Z     self.run_test(test_name, pipe)
2022-11-23T03:00:27.9847819Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:27.9848177Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:27.9848872Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:27.9849324Z     getattr(self, test_name)()
2022-11-23T03:00:27.9849809Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:27.9850169Z     self.run_test(test_name, pipe)
2022-11-23T03:00:27.9850868Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:27.9851230Z     fn()
2022-11-23T03:00:27.9851698Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:27.9852088Z     getattr(self, test_name)()
2022-11-23T03:00:27.9852618Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:27.9853001Z     test(self, **param_kwargs)
2022-11-23T03:00:27.9853528Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:27.9853905Z     fn()
2022-11-23T03:00:27.9854389Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:27.9855084Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9855610Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:27.9856020Z     test(self, **param_kwargs)
2022-11-23T03:00:27.9856397Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:27.9856989Z     self.run_subtests(
2022-11-23T03:00:27.9857512Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:27.9857912Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9858407Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:27.9858843Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:27.9859282Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:27.9859637Z     self.run_subtests(
2022-11-23T03:00:27.9860153Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:27.9860579Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:27.9861116Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:27.9861526Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:27.9862172Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:27.9862581Z     output = model(*input)
2022-11-23T03:00:27.9863082Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:27.9863510Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:27.9864019Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:27.9864569Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:27.9865079Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:27.9865468Z     output = model(*input)
2022-11-23T03:00:27.9865978Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:27.9866401Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:27.9867110Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:27.9867501Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:27.9868133Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:27.9868505Z     _lazy_init(state, module)
2022-11-23T03:00:27.9869043Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:27.9869506Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:27.9870026Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:27.9870439Z     handle.init_flat_param_attributes()
2022-11-23T03:00:27.9870987Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:27.9871385Z     _lazy_init(state, module)
2022-11-23T03:00:27.9871865Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:27.9872413Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9872917Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:27.9873295Z     handle.init_flat_param_attributes()
2022-11-23T03:00:27.9873830Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:27.9874210Z     p_assert(
2022-11-23T03:00:27.9874829Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:27.9875204Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9875701Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:27.9876086Z     traceback.print_stack()
2022-11-23T03:00:27.9876610Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:27.9876999Z     p_assert(
2022-11-23T03:00:27.9877476Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:27.9877839Z     traceback.print_stack()
2022-11-23T03:00:27.9878264Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:27.9878745Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:27.9879232Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:27.9879704Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:27.9880077Z   File "<string>", line 1, in <module>
2022-11-23T03:00:27.9880360Z   File "<string>", line 1, in <module>
2022-11-23T03:00:27.9880770Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:27.9881150Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:27.9881522Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:27.9881872Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:27.9882399Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:27.9882752Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:27.9883116Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:27.9883419Z     self.run()
2022-11-23T03:00:27.9883736Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:27.9884092Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:27.9884437Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:27.9884785Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:27.9885155Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:27.9885515Z     self.run()
2022-11-23T03:00:27.9885977Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:27.9886349Z     self.run_test(test_name, pipe)
2022-11-23T03:00:27.9886698Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:27.9887031Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:27.9887545Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:27.9887920Z     getattr(self, test_name)()
2022-11-23T03:00:27.9888377Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:27.9888742Z     self.run_test(test_name, pipe)
2022-11-23T03:00:27.9889245Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:27.9889576Z     fn()
2022-11-23T03:00:27.9890221Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:27.9890610Z     getattr(self, test_name)()
2022-11-23T03:00:27.9891126Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:27.9891499Z     test(self, **param_kwargs)
2022-11-23T03:00:27.9892006Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:27.9892425Z     fn()
2022-11-23T03:00:27.9892887Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:27.9893280Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9893800Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:27.9894191Z     test(self, **param_kwargs)
2022-11-23T03:00:27.9894577Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:27.9894952Z     self.run_subtests(
2022-11-23T03:00:27.9895452Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:27.9895974Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9896640Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:27.9897273Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:27.9897707Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:27.9898061Z     self.run_subtests(
2022-11-23T03:00:27.9898662Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:27.9899088Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:27.9899607Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:27.9900176Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:27.9900915Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:27.9901294Z     output = model(*input)
2022-11-23T03:00:27.9901789Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:27.9902203Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:27.9902701Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:27.9903065Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:27.9903758Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:27.9904208Z     output = model(*input)
2022-11-23T03:00:27.9904704Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:27.9905146Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:27.9905645Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:27.9906029Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:27.9906533Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:27.9906917Z     _lazy_init(state, module)
2022-11-23T03:00:27.9907434Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:27.9908052Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:27.9908591Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:27.9909010Z     handle.init_flat_param_attributes()
2022-11-23T03:00:27.9909547Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:27.9909918Z     _lazy_init(state, module)
2022-11-23T03:00:27.9910458Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:27.9910983Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9911616Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:27.9912030Z     handle.init_flat_param_attributes()
2022-11-23T03:00:27.9912588Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:27.9912950Z     p_assert(
2022-11-23T03:00:27.9913407Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:27.9913783Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9914271Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:27.9914631Z     traceback.print_stack()
2022-11-23T03:00:27.9915163Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:27.9915539Z     p_assert(
2022-11-23T03:00:27.9915999Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:27.9916360Z     traceback.print_stack()
2022-11-23T03:00:27.9916642Z   File "<string>", line 1, in <module>
2022-11-23T03:00:27.9917066Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:27.9917428Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:27.9917954Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:27.9918311Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:27.9918671Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:27.9918999Z     self.run()
2022-11-23T03:00:27.9919319Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:27.9919671Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:27.9920150Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:27.9920521Z     self.run_test(test_name, pipe)
2022-11-23T03:00:27.9921025Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:27.9921383Z     getattr(self, test_name)()
2022-11-23T03:00:27.9921882Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:27.9922294Z     fn()
2022-11-23T03:00:27.9922928Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:27.9923320Z     test(self, **param_kwargs)
2022-11-23T03:00:27.9923829Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:27.9924218Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9924606Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:27.9924983Z     self.run_subtests(
2022-11-23T03:00:27.9925485Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:27.9925892Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:27.9926600Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:27.9926756Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:27.9927126Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:27.9927247Z     output = model(*input)
2022-11-23T03:00:27.9927563Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:27.9927703Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:27.9928049Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:27.9928222Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:27.9928583Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:27.9928703Z     _lazy_init(state, module)
2022-11-23T03:00:27.9929052Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:27.9929197Z     handle.init_flat_param_attributes()
2022-11-23T03:00:27.9929526Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:27.9929649Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9929997Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:27.9930097Z     p_assert(
2022-11-23T03:00:27.9930429Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:27.9930558Z     traceback.print_stack()
2022-11-23T03:00:27.9930686Z   File "<string>", line 1, in <module>
2022-11-23T03:00:27.9931119Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:27.9931276Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:27.9931466Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:27.9931683Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:27.9931906Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:27.9932012Z     self.run()
2022-11-23T03:00:27.9932216Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:27.9932364Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:27.9932717Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:27.9932855Z     self.run_test(test_name, pipe)
2022-11-23T03:00:27.9933202Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:27.9933335Z     getattr(self, test_name)()
2022-11-23T03:00:27.9933697Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:27.9933853Z     fn()
2022-11-23T03:00:27.9934227Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:27.9934354Z     test(self, **param_kwargs)
2022-11-23T03:00:27.9934714Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:27.9934841Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9935074Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:27.9935188Z     self.run_subtests(
2022-11-23T03:00:27.9935542Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:27.9935713Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:27.9936083Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:27.9936241Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:27.9936622Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:27.9936930Z     output = model(*input)
2022-11-23T03:00:27.9937256Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:27.9937401Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:27.9937781Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:27.9937964Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:27.9938340Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:27.9938463Z     _lazy_init(state, module)
2022-11-23T03:00:27.9938822Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:27.9938968Z     handle.init_flat_param_attributes()
2022-11-23T03:00:27.9939287Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:27.9939414Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9939794Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:27.9939905Z     p_assert(
2022-11-23T03:00:27.9940246Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:27.9940375Z     traceback.print_stack()
2022-11-23T03:00:27.9940616Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:27.9940933Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:27.9941158Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:27.9941395Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:27.9941532Z   File "<string>", line 1, in <module>
2022-11-23T03:00:27.9941663Z   File "<string>", line 1, in <module>
2022-11-23T03:00:27.9941879Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:27.9942025Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:27.9942233Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:27.9942385Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:27.9942579Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:27.9942723Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:27.9942946Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:27.9943114Z     self.run()
2022-11-23T03:00:27.9943348Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:27.9943474Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:27.9943679Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:27.9943810Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:27.9944161Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:27.9944298Z     self.run_test(test_name, pipe)
2022-11-23T03:00:27.9944516Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:27.9944622Z     self.run()
2022-11-23T03:00:27.9944993Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:27.9945119Z     getattr(self, test_name)()
2022-11-23T03:00:27.9945324Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:27.9945456Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:27.9945824Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:27.9946078Z     fn()
2022-11-23T03:00:27.9946411Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:27.9946542Z     self.run_test(test_name, pipe)
2022-11-23T03:00:27.9946903Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:27.9947019Z     test(self, **param_kwargs)
2022-11-23T03:00:27.9947366Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:27.9947471Z     getattr(self, test_name)()
2022-11-23T03:00:27.9947825Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:27.9947951Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9948300Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:27.9948398Z     fn()
2022-11-23T03:00:27.9948640Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:27.9948753Z     self.run_subtests(
2022-11-23T03:00:27.9949086Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:27.9949210Z     test(self, **param_kwargs)
2022-11-23T03:00:27.9949725Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:27.9949973Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:27.9950344Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:27.9950476Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9950844Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:27.9950999Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:27.9951230Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:27.9951346Z     self.run_subtests(
2022-11-23T03:00:27.9951726Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:27.9951847Z     output = model(*input)
2022-11-23T03:00:27.9952205Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:27.9952373Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:27.9952757Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:27.9952900Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:27.9953246Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:27.9953403Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:27.9953936Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:27.9954111Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:27.9954479Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:27.9954599Z     output = model(*input)
2022-11-23T03:00:27.9954956Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:27.9955081Z     _lazy_init(state, module)
2022-11-23T03:00:27.9955582Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:27.9955704Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:27.9956055Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:27.9956203Z     handle.init_flat_param_attributes()
2022-11-23T03:00:27.9956583Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:27.9956761Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:27.9957103Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:27.9957230Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9957606Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:27.9957712Z     _lazy_init(state, module)
2022-11-23T03:00:27.9958248Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:27.9958349Z     p_assert(
2022-11-23T03:00:27.9958692Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:27.9958838Z     handle.init_flat_param_attributes()
2022-11-23T03:00:27.9959166Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:27.9959297Z     traceback.print_stack()
2022-11-23T03:00:27.9959625Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:27.9959727Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9960326Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:27.9960440Z     p_assert(
2022-11-23T03:00:27.9960776Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:27.9960905Z     traceback.print_stack()
2022-11-23T03:00:27.9961036Z   File "<string>", line 1, in <module>
2022-11-23T03:00:27.9961166Z   File "<string>", line 1, in <module>
2022-11-23T03:00:27.9961358Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:27.9961507Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:27.9961712Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:27.9961865Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:27.9962078Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:27.9962225Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:27.9962440Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:27.9962602Z     self.run()
2022-11-23T03:00:27.9962786Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:27.9962938Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:27.9963144Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:27.9963295Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:27.9963507Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:27.9963612Z     self.run()
2022-11-23T03:00:27.9963963Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:27.9964078Z     self.run_test(test_name, pipe)
2022-11-23T03:00:27.9964285Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:27.9964435Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:27.9964964Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:27.9965092Z     getattr(self, test_name)()
2022-11-23T03:00:27.9965421Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:27.9965552Z     self.run_test(test_name, pipe)
2022-11-23T03:00:27.9966066Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:27.9966147Z     fn()
2022-11-23T03:00:27.9966512Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:27.9966636Z     getattr(self, test_name)()
2022-11-23T03:00:27.9967005Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:27.9967133Z     test(self, **param_kwargs)
2022-11-23T03:00:27.9967495Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:27.9967597Z     fn()
2022-11-23T03:00:27.9967959Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:27.9968068Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9968440Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:27.9968563Z     test(self, **param_kwargs)
2022-11-23T03:00:27.9968815Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:27.9969078Z     self.run_subtests(
2022-11-23T03:00:27.9969428Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:27.9969601Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9969953Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:27.9970096Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:27.9970340Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:27.9970457Z     self.run_subtests(
2022-11-23T03:00:27.9970812Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:27.9970963Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:27.9971305Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:27.9971467Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:27.9971836Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:27.9971934Z     output = model(*input)
2022-11-23T03:00:27.9972527Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:27.9972688Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:27.9973021Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:27.9973164Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:27.9973544Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:27.9973667Z     output = model(*input)
2022-11-23T03:00:27.9974044Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:27.9974200Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:27.9974537Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:27.9974688Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:27.9975058Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:27.9975181Z     _lazy_init(state, module)
2022-11-23T03:00:27.9975556Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:27.9975739Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:27.9976093Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:27.9976218Z     handle.init_flat_param_attributes()
2022-11-23T03:00:27.9976589Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:27.9976901Z     _lazy_init(state, module)
2022-11-23T03:00:27.9977264Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:27.9977399Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9977754Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:27.9977899Z     handle.init_flat_param_attributes()
2022-11-23T03:00:27.9978280Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:27.9978365Z     p_assert(
2022-11-23T03:00:27.9978706Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:27.9978835Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9979177Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:27.9979383Z     traceback.print_stack()
2022-11-23T03:00:27.9979781Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:27.9979889Z     p_assert(
2022-11-23T03:00:27.9980226Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:27.9980332Z     traceback.print_stack()
2022-11-23T03:00:27.9980574Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:27.9980816Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:27.9981055Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:27.9981291Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:27.9981424Z   File "<string>", line 1, in <module>
2022-11-23T03:00:27.9981555Z   File "<string>", line 1, in <module>
2022-11-23T03:00:27.9981773Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:27.9981970Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:27.9982176Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:27.9982329Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:27.9982693Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:27.9982833Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:27.9983041Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:27.9983144Z     self.run()
2022-11-23T03:00:27.9983320Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:27.9983470Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:27.9983670Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:27.9983823Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:27.9984030Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:27.9984136Z     self.run()
2022-11-23T03:00:27.9984482Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:27.9984617Z     self.run_test(test_name, pipe)
2022-11-23T03:00:27.9984794Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:27.9984939Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:27.9985293Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:27.9985422Z     getattr(self, test_name)()
2022-11-23T03:00:27.9985753Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:27.9985885Z     self.run_test(test_name, pipe)
2022-11-23T03:00:27.9986242Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:27.9986342Z     fn()
2022-11-23T03:00:27.9986671Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:27.9986796Z     getattr(self, test_name)()
2022-11-23T03:00:27.9987152Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:27.9987273Z     test(self, **param_kwargs)
2022-11-23T03:00:27.9987620Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:27.9987718Z     fn()
2022-11-23T03:00:27.9988067Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:27.9988170Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9988578Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:27.9988710Z     test(self, **param_kwargs)
2022-11-23T03:00:27.9988955Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:27.9989068Z     self.run_subtests(
2022-11-23T03:00:27.9989420Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:27.9989543Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9989884Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:27.9990023Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:27.9990266Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:27.9990379Z     self.run_subtests(
2022-11-23T03:00:27.9990926Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:27.9991168Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:27.9991520Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:27.9991684Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:27.9992063Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:27.9992235Z     output = model(*input)
2022-11-23T03:00:27.9992585Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:27.9992741Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:27.9993071Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:27.9993218Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:27.9993597Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:27.9993720Z     output = model(*input)
2022-11-23T03:00:27.9994102Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:27.9994280Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:27.9994588Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:27.9994730Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:27.9995103Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:27.9995228Z     _lazy_init(state, module)
2022-11-23T03:00:27.9995607Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:27.9995784Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:27.9996141Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:27.9996286Z     handle.init_flat_param_attributes()
2022-11-23T03:00:27.9996633Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:27.9996755Z     _lazy_init(state, module)
2022-11-23T03:00:27.9997100Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:27.9997228Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9997581Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:27.9997728Z     handle.init_flat_param_attributes()
2022-11-23T03:00:27.9998162Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:27.9998278Z     p_assert(
2022-11-23T03:00:27.9998601Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:27.9998727Z     return func(*args, **kwargs)
2022-11-23T03:00:27.9999065Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:27.9999198Z     traceback.print_stack()
2022-11-23T03:00:27.9999574Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:27.9999832Z     p_assert(
2022-11-23T03:00:28.0000158Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0000282Z     traceback.print_stack()
2022-11-23T03:00:28.0000389Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0000520Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0000726Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0001106Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0001301Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0001455Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0001670Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0001792Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0002009Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0002115Z     self.run()
2022-11-23T03:00:28.0002317Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0002468Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0002675Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0002827Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0003022Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0003133Z     self.run()
2022-11-23T03:00:28.0003489Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0003625Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0003983Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0004127Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0004480Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0004603Z     getattr(self, test_name)()
2022-11-23T03:00:28.0004910Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0005035Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0005394Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0005497Z     fn()
2022-11-23T03:00:28.0005849Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0005970Z     getattr(self, test_name)()
2022-11-23T03:00:28.0006326Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0006448Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0006777Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0006873Z     fn()
2022-11-23T03:00:28.0007217Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0007342Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0007749Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0007884Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0008308Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:28.0008427Z     self.run_subtests(
2022-11-23T03:00:28.0008769Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0008897Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0009251Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0009415Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0009669Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:28.0009787Z     self.run_subtests(
2022-11-23T03:00:28.0010162Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0010426Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0010768Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0011086Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0011457Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0011577Z     output = model(*input)
2022-11-23T03:00:28.0011934Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0012084Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0012402Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0012545Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0012893Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0013019Z     output = model(*input)
2022-11-23T03:00:28.0013389Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0013564Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0013882Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0014022Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0014380Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0014498Z     _lazy_init(state, module)
2022-11-23T03:00:28.0014846Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0015019Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0015374Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0015516Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0015874Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0015993Z     _lazy_init(state, module)
2022-11-23T03:00:28.0016322Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0016446Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0017149Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0017292Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0017766Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0017891Z     p_assert(
2022-11-23T03:00:28.0018234Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0018361Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0018702Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0018832Z     traceback.print_stack()
2022-11-23T03:00:28.0019190Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0019293Z     p_assert(
2022-11-23T03:00:28.0019624Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0019751Z     traceback.print_stack()
2022-11-23T03:00:28.0019997Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0020235Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0020689Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0020917Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0021025Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0021151Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0021362Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0021506Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0021706Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0021857Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0022063Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0022205Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0022394Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0022500Z     self.run()
2022-11-23T03:00:28.0022697Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0023020Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0023225Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0023390Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0023594Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0023679Z     self.run()
2022-11-23T03:00:28.0024031Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0024166Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0024376Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0024532Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0024905Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0025030Z     getattr(self, test_name)()
2022-11-23T03:00:28.0025364Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0025478Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0025847Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0025944Z     fn()
2022-11-23T03:00:28.0026307Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0026431Z     getattr(self, test_name)()
2022-11-23T03:00:28.0027002Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0027129Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0027482Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0027562Z     fn()
2022-11-23T03:00:28.0027911Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0028037Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0028393Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0028510Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0028750Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:28.0028865Z     self.run_subtests(
2022-11-23T03:00:28.0029195Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0029322Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0029673Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0029888Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0030133Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:28.0030246Z     self.run_subtests(
2022-11-23T03:00:28.0030785Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0030940Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0031294Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0031437Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0031820Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0031943Z     output = model(*input)
2022-11-23T03:00:28.0032313Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0032474Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0032801Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0032945Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0033482Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0033584Z     output = model(*input)
2022-11-23T03:00:28.0034124Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0034306Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0034640Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0034786Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0035158Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0035286Z     _lazy_init(state, module)
2022-11-23T03:00:28.0035662Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0035818Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0036173Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0036321Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0036692Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0036857Z     _lazy_init(state, module)
2022-11-23T03:00:28.0037212Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0037333Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0037691Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0037815Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0038200Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0038305Z     p_assert(
2022-11-23T03:00:28.0038651Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0038781Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0039118Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0039251Z     traceback.print_stack()
2022-11-23T03:00:28.0039628Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0039767Z     p_assert(
2022-11-23T03:00:28.0040111Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0040242Z     traceback.print_stack()
2022-11-23T03:00:28.0040376Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0040506Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0040720Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0040867Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0041051Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0041363Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0041569Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0041713Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0041927Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0042031Z     self.run()
2022-11-23T03:00:28.0042408Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0042574Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0042746Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0042895Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0043109Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0043216Z     self.run()
2022-11-23T03:00:28.0043564Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0043704Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0043915Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0044042Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0044413Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0044538Z     getattr(self, test_name)()
2022-11-23T03:00:28.0044879Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0045014Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0045377Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0045477Z     fn()
2022-11-23T03:00:28.0045840Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0045944Z     getattr(self, test_name)()
2022-11-23T03:00:28.0046518Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0046650Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0047009Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0047106Z     fn()
2022-11-23T03:00:28.0047450Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0047573Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0047926Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0048025Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0048271Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:28.0048382Z     self.run_subtests(
2022-11-23T03:00:28.0048737Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0048859Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0049260Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0049420Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0049662Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:28.0049753Z     self.run_subtests(
2022-11-23T03:00:28.0050107Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0050255Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0050592Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0050751Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0051296Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0051422Z     output = model(*input)
2022-11-23T03:00:28.0051795Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0051928Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0052258Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0052403Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0052784Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0052911Z     output = model(*input)
2022-11-23T03:00:28.0053292Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0053474Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0053959Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0054081Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0054442Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0054560Z     _lazy_init(state, module)
2022-11-23T03:00:28.0054923Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0055096Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0055438Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0055748Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0056167Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0056276Z     _lazy_init(state, module)
2022-11-23T03:00:28.0056627Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0056938Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0057312Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0057458Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0057844Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0057948Z     p_assert(
2022-11-23T03:00:28.0058289Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0058546Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0058877Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0059009Z     traceback.print_stack()
2022-11-23T03:00:28.0059376Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0059555Z     p_assert(
2022-11-23T03:00:28.0059887Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0060013Z     traceback.print_stack()
2022-11-23T03:00:28.0060248Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0060660Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0060870Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0061107Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0061236Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0061454Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0061601Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0061807Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0061960Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0062070Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0062286Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0062394Z     self.run()
2022-11-23T03:00:28.0062599Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0062747Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0062960Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0063104Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0063438Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0063577Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0063789Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0063943Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0064316Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0064444Z     getattr(self, test_name)()
2022-11-23T03:00:28.0064659Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0064766Z     self.run()
2022-11-23T03:00:28.0065108Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0065204Z     fn()
2022-11-23T03:00:28.0065411Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0065712Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0066132Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0066266Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0066599Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0066730Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0067058Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0067183Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0067532Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0067658Z     getattr(self, test_name)()
2022-11-23T03:00:28.0067906Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:28.0068021Z     self.run_subtests(
2022-11-23T03:00:28.0068375Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0068711Z     fn()
2022-11-23T03:00:28.0069051Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0069220Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0069587Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0069713Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0070071Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0070231Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0070589Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0070719Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0071077Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0071203Z     output = model(*input)
2022-11-23T03:00:28.0071609Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:28.0071724Z     self.run_subtests(
2022-11-23T03:00:28.0072046Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0072185Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0072528Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0072689Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0073037Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0073216Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0073572Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0073722Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0074079Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0074198Z     _lazy_init(state, module)
2022-11-23T03:00:28.0074561Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0074679Z     output = model(*input)
2022-11-23T03:00:28.0075181Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0075328Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0075707Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0075859Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0076206Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0076336Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0076714Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0076894Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0077257Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0077365Z     p_assert(
2022-11-23T03:00:28.0077737Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0077864Z     _lazy_init(state, module)
2022-11-23T03:00:28.0078205Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0078387Z     traceback.print_stack()
2022-11-23T03:00:28.0078748Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0078895Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0079217Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0079345Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0079732Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0079839Z     p_assert(
2022-11-23T03:00:28.0080174Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0080304Z     traceback.print_stack()
2022-11-23T03:00:28.0080439Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0080633Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0080789Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0080996Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0081161Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0081429Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0081537Z     self.run()
2022-11-23T03:00:28.0081745Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0081894Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0082219Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0082354Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0082722Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0082999Z     getattr(self, test_name)()
2022-11-23T03:00:28.0083353Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0083451Z     fn()
2022-11-23T03:00:28.0083810Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0083933Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0084256Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0084385Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0084631Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:28.0084746Z     self.run_subtests(
2022-11-23T03:00:28.0085091Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0085319Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0085687Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0085836Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0086179Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0086296Z     output = model(*input)
2022-11-23T03:00:28.0086612Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0086751Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0087114Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0087291Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0087649Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0087824Z     _lazy_init(state, module)
2022-11-23T03:00:28.0088149Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0088290Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0088622Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0088745Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0089117Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0089218Z     p_assert(
2022-11-23T03:00:28.0089543Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0089672Z     traceback.print_stack()
2022-11-23T03:00:28.0089779Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0089992Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0090136Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0090338Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0090486Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0090696Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0090799Z     self.run()
2022-11-23T03:00:28.0091161Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0091322Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0091663Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0091798Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0092167Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0092339Z     getattr(self, test_name)()
2022-11-23T03:00:28.0092710Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0092810Z     fn()
2022-11-23T03:00:28.0093157Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0093281Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0093643Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0093771Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0094025Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:28.0094140Z     self.run_subtests(
2022-11-23T03:00:28.0094546Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0094717Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0095069Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0095223Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0095600Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0095723Z     output = model(*input)
2022-11-23T03:00:28.0096054Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0096199Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0096578Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0096939Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0097307Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0097514Z     _lazy_init(state, module)
2022-11-23T03:00:28.0097872Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0098020Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0098364Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0098491Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0098869Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0098981Z     p_assert(
2022-11-23T03:00:28.0099299Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0099428Z     traceback.print_stack()
2022-11-23T03:00:28.0099675Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0099916Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0100152Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0100386Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0100517Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0100801Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0100989Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0101129Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0101499Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0101650Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0101860Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0102000Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0102217Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0102318Z     self.run()
2022-11-23T03:00:28.0102499Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0102647Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0102841Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0102990Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0103202Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0103308Z     self.run()
2022-11-23T03:00:28.0103659Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0103773Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0104201Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0104355Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0104717Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0104841Z     getattr(self, test_name)()
2022-11-23T03:00:28.0105165Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0105281Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0105635Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0105713Z     fn()
2022-11-23T03:00:28.0106064Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0106189Z     getattr(self, test_name)()
2022-11-23T03:00:28.0106545Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0106667Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0107072Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0107173Z     fn()
2022-11-23T03:00:28.0107514Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0107616Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0107973Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0108093Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0108338Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:28.0108622Z     self.run_subtests(
2022-11-23T03:00:28.0108989Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0109114Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0109474Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0109616Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0109868Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:28.0109984Z     self.run_subtests(
2022-11-23T03:00:28.0110355Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0110566Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0110929Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0111103Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0111645Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0111935Z     output = model(*input)
2022-11-23T03:00:28.0112283Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0112439Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0112788Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0112932Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0113296Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0113414Z     output = model(*input)
2022-11-23T03:00:28.0113788Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0114002Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0114340Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0114481Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0114847Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0114966Z     _lazy_init(state, module)
2022-11-23T03:00:28.0115337Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0115509Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0115890Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0116019Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0116375Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0116483Z     _lazy_init(state, module)
2022-11-23T03:00:28.0116861Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0116975Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0117314Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0117446Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0117812Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0117895Z     p_assert(
2022-11-23T03:00:28.0118222Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0118334Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0118813Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0118929Z     traceback.print_stack()
2022-11-23T03:00:28.0119288Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0119378Z     p_assert(
2022-11-23T03:00:28.0119690Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0119792Z     traceback.print_stack()
2022-11-23T03:00:28.0119906Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0120098Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0120225Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0120409Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0120545Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0120741Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0120827Z     self.run()
2022-11-23T03:00:28.0121012Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0121145Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0121462Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0121580Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0121917Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0122025Z     getattr(self, test_name)()
2022-11-23T03:00:28.0122364Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0122440Z     fn()
2022-11-23T03:00:28.0122784Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0122890Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0123460Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0123571Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0123809Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:28.0123911Z     self.run_subtests(
2022-11-23T03:00:28.0124279Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0124400Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0124754Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0124896Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0125259Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0125372Z     output = model(*input)
2022-11-23T03:00:28.0125688Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0125874Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0126242Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0126399Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0126754Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0127021Z     _lazy_init(state, module)
2022-11-23T03:00:28.0127137Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0127467Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0127594Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0127913Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0128025Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0128210Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0128338Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0128871Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0128963Z     p_assert(
2022-11-23T03:00:28.0129154Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0129295Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0129620Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0129733Z     traceback.print_stack()
2022-11-23T03:00:28.0129929Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0130022Z     self.run()
2022-11-23T03:00:28.0130216Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0130353Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0130680Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0130800Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0131151Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0131256Z     getattr(self, test_name)()
2022-11-23T03:00:28.0131760Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0131842Z     fn()
2022-11-23T03:00:28.0132180Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0132286Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0132669Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0132785Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0133018Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:28.0133108Z     self.run_subtests(
2022-11-23T03:00:28.0133440Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0133585Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0133921Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0134056Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0134589Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0134688Z     output = model(*input)
2022-11-23T03:00:28.0135005Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0135195Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0135563Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0135729Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0136086Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0136193Z     _lazy_init(state, module)
2022-11-23T03:00:28.0136531Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0136663Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0137206Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0137313Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0137687Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0137777Z     p_assert(
2022-11-23T03:00:28.0138103Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0138217Z     traceback.print_stack()
2022-11-23T03:00:28.0138444Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0138670Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0138885Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0139101Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0139222Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0139424Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0139559Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0139750Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0139890Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0140097Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0140187Z     self.run()
2022-11-23T03:00:28.0140371Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0140509Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0140841Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0140965Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0141581Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0141700Z     getattr(self, test_name)()
2022-11-23T03:00:28.0142046Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0142128Z     fn()
2022-11-23T03:00:28.0142633Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0142747Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0142863Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0143213Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0143325Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0143561Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:28.0143662Z     self.run_subtests(
2022-11-23T03:00:28.0143857Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0143988Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0144406Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0144555Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0144746Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0144884Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0145237Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0145376Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0145573Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0145666Z     self.run()
2022-11-23T03:00:28.0146036Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0146144Z     output = model(*input)
2022-11-23T03:00:28.0146341Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0146474Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0146939Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0147064Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0147374Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0147492Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0147845Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0148002Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0148343Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0148451Z     getattr(self, test_name)()
2022-11-23T03:00:28.0148799Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0148903Z     _lazy_init(state, module)
2022-11-23T03:00:28.0149232Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0149317Z     fn()
2022-11-23T03:00:28.0149643Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0149770Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0150111Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0150219Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0150595Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0150712Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0151237Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0151339Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0151705Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0151796Z     p_assert(
2022-11-23T03:00:28.0152036Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:28.0152137Z     self.run_subtests(
2022-11-23T03:00:28.0152465Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0152580Z     traceback.print_stack()
2022-11-23T03:00:28.0152920Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0153068Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0153482Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0153625Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0153987Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0154094Z     output = model(*input)
2022-11-23T03:00:28.0154409Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0154539Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0155054Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0155216Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0155563Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0155669Z     _lazy_init(state, module)
2022-11-23T03:00:28.0156169Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0156300Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0156629Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0156742Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0157103Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0157193Z     p_assert(
2022-11-23T03:00:28.0157516Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0157629Z     traceback.print_stack()
2022-11-23T03:00:28.0157748Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0157948Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0158081Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0158273Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0158406Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0158607Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0158698Z     self.run()
2022-11-23T03:00:28.0159038Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0159168Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0159486Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0159601Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0160040Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0160159Z     getattr(self, test_name)()
2022-11-23T03:00:28.0160504Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0160586Z     fn()
2022-11-23T03:00:28.0161108Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0161218Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0161564Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0161678Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0161909Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:28.0162008Z     self.run_subtests(
2022-11-23T03:00:28.0162355Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0162508Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0162922Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0163060Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0163424Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0163530Z     output = model(*input)
2022-11-23T03:00:28.0163835Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0163965Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0164329Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0164493Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0164854Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0164964Z     _lazy_init(state, module)
2022-11-23T03:00:28.0165305Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0165599Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0165906Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0166016Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0166369Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0166456Z     p_assert(
2022-11-23T03:00:28.0166770Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0166879Z     traceback.print_stack()
2022-11-23T03:00:28.0167166Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0167368Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0167496Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0167688Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0167827Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0168028Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0168117Z     self.run()
2022-11-23T03:00:28.0168309Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0168443Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0168768Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0168888Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0169289Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0169408Z     getattr(self, test_name)()
2022-11-23T03:00:28.0169761Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0169845Z     fn()
2022-11-23T03:00:28.0170350Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0170459Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0170788Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0170898Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0171129Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step
2022-11-23T03:00:28.0171227Z     self.run_subtests(
2022-11-23T03:00:28.0171560Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0171703Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0172101Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0172236Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0172580Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0172685Z     output = model(*input)
2022-11-23T03:00:28.0172990Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0173115Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0173467Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0173622Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0173970Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0174077Z     _lazy_init(state, module)
2022-11-23T03:00:28.0174399Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0174524Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0174840Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0174950Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0175473Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0175574Z     p_assert(
2022-11-23T03:00:28.0175893Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0176006Z     traceback.print_stack()
2022-11-23T03:00:28.0176233Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0176463Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0176687Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0177095Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0177312Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0177528Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0177745Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0177958Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0178250Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0178468Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0178689Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0178903Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0179119Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0179335Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0179551Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0179762Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0179977Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0180189Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0180405Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0180685Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0180893Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0181107Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0181321Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0181536Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0182547Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T03:00:28.0182773Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-11-23T03:00:28.0183754Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T03:00:28.0183973Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-11-23T03:00:28.0184955Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T03:00:28.0185176Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-11-23T03:00:28.0186300Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T03:00:28.0186554Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-11-23T03:00:28.0186778Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0186999Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0187210Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0187422Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0187629Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0187836Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0188043Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0188255Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0188461Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0188717Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0188922Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0189129Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0189336Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0189540Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0189743Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0189948Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0190149Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0190359Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0190567Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0190775Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0190981Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0191365Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0191582Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0191806Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0192001Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0192219Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0192481Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0192701Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0192913Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0193124Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0193337Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0193551Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0193764Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0193970Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0194238Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0194459Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0194672Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0194886Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0195098Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0195315Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0195528Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0195736Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0195948Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0196165Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0196427Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0196639Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0196853Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0197068Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0197814Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T03:00:28.0198533Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T03:00:28.0199267Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T03:00:28.0199983Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T03:00:28.0200206Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0200577Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0200785Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0200994Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0201203Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0201410Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0201789Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0202003Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0202220Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0202483Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0202700Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0202915Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0203128Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0203343Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0203557Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0203771Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0203982Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0204195Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0204556Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0204766Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0205023Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0205228Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0205430Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0205633Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0205734Z dist init r=1, world=4
2022-11-23T03:00:28.0206043Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.0206342Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.0206652Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.0206949Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.0207237Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.0207538Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.0207833Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.0208122Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.0208428Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.0208887Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.0209187Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.0209494Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.0209595Z dist init r=2, world=4
2022-11-23T03:00:28.0209946Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.0210259Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.0210625Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.0210926Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.0211219Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.0211526Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.0212027Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.0212310Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.0212606Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.0212896Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.0213185Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.0213479Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.0213572Z dist init r=0, world=4
2022-11-23T03:00:28.0213857Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.0214140Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.0214429Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.0214716Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.0214999Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.0215478Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.0215777Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.0216072Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.0216376Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.0216899Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.0217215Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.0217520Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.0217619Z dist init r=3, world=4
2022-11-23T03:00:28.0217913Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.0218360Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.0218659Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.0219021Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.0219306Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.0219598Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.0219884Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.0220171Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.0220463Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.0220747Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.0221029Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.0221318Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.0221406Z ok (21.156s)
2022-11-23T03:00:28.0221592Z   test_delayed_optim_step_offload_true_none (__main__.TestParityWithDDP)
2022-11-23T03:00:28.0222088Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.0222291Z   test_delayed_optim_step_offload_true_shard_grad_op (__main__.TestParityWithDDP)
2022-11-23T03:00:28.0222770Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.0222967Z   test_delayed_reduce_scatter_offload_false_no_shard (__main__.TestParityWithDDP)
2022-11-23T03:00:28.0223429Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.0223785Z   test_delayed_reduce_scatter_offload_false_none (__main__.TestParityWithDDP)
2022-11-23T03:00:28.0224148Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 123154
2022-11-23T03:00:28.0224363Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 123155
2022-11-23T03:00:28.0224594Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 123156
2022-11-23T03:00:28.0224778Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 123157
2022-11-23T03:00:28.0225147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:00:28.0225312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:00:28.0225682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:00:28.0225857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:00:28.0226215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:00:28.0226379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:00:28.0226798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:00:28.0226975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:00:28.0227476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:00:28.0227633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:00:28.0227986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:00:28.0228157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:00:28.0228486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:00:28.0228643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:00:28.0228991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:00:28.0229166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:00:28.0229392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:00:28.0229616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:00:28.0229837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:00:28.0230054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:00:28.0230431Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:00:28.0230801Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:00:28.0231346Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:00:28.0231785Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:00:28.0232010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:00:28.0232221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:00:28.0232434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:00:28.0232649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:00:28.0232871Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0233148Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0233369Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0233592Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0234923Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:00:28.0235018Z   warnings.warn(
2022-11-23T03:00:28.0236008Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:00:28.0236174Z   warnings.warn(
2022-11-23T03:00:28.0237155Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:00:28.0237253Z   warnings.warn(
2022-11-23T03:00:28.0238228Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:00:28.0238329Z   warnings.warn(
2022-11-23T03:00:28.0238551Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0238774Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0238992Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0239203Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0239418Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0239634Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0239855Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0240068Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0240283Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0240496Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0240724Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0240918Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0241136Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0241398Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0241616Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0241831Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0242045Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0242259Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0242474Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0242688Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0242893Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0243106Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0243321Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0243538Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0243802Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0244012Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0244225Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0244438Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0244644Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0244856Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0245066Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0245280Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0245499Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0245713Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0245925Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0246136Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0246888Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T03:00:28.0247768Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T03:00:28.0248464Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T03:00:28.0249165Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T03:00:28.0249373Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0249633Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0250026Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0250239Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0250458Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0250673Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0250888Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0251105Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0251318Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0251530Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0251746Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0252014Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0252231Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0252446Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0252660Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0252872Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0253085Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0253293Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0253511Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0253725Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0253938Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0254303Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0254508Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0254713Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0254923Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0255131Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0255337Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0255543Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0255748Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0255956Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0256159Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0256540Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0256947Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0257173Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0257385Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0257591Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0257870Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0258095Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0258310Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0258523Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0258733Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0258945Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0259313Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0259513Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0259724Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0259931Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0260197Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0260402Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0261292Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T03:00:28.0262018Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T03:00:28.0262748Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T03:00:28.0263470Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T03:00:28.0263688Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0263907Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0264124Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0264337Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0264552Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0264769Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0264980Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0265191Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0265291Z dist init r=0, world=4
2022-11-23T03:00:28.0265388Z dist init r=3, world=4
2022-11-23T03:00:28.0265483Z dist init r=1, world=4
2022-11-23T03:00:28.0265572Z dist init r=2, world=4
2022-11-23T03:00:28.0265660Z ok (53.823s)
2022-11-23T03:00:28.0266028Z   test_delayed_reduce_scatter_offload_false_shard_grad_op (__main__.TestParityWithDDP)
2022-11-23T03:00:28.0266552Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 123455
2022-11-23T03:00:28.0266779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 123456
2022-11-23T03:00:28.0266984Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 123457
2022-11-23T03:00:28.0267190Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 123458
2022-11-23T03:00:28.0267557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:00:28.0267716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:00:28.0268084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:00:28.0268266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:00:28.0268621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:00:28.0268784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:00:28.0269388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:00:28.0269562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:00:28.0269898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:00:28.0270052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:00:28.0270391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:00:28.0270561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:00:28.0270912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:00:28.0271066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:00:28.0271420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:00:28.0271593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:00:28.0271820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:00:28.0272042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:00:28.0272257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:00:28.0272475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:00:28.0273035Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:00:28.0273420Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:00:28.0273810Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:00:28.0274189Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:00:28.0274408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:00:28.0274623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:00:28.0274837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:00:28.0275041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:00:28.0275311Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0275540Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0275762Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0275985Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0276994Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:00:28.0277095Z   warnings.warn(
2022-11-23T03:00:28.0278081Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:00:28.0278227Z   warnings.warn(
2022-11-23T03:00:28.0279206Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:00:28.0279304Z   warnings.warn(
2022-11-23T03:00:28.0280275Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:00:28.0280374Z   warnings.warn(
2022-11-23T03:00:28.0280596Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0280810Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0281028Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0281250Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0281470Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0281682Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0281901Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0282115Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0282330Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0282539Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0282750Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0282964Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0283178Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0283589Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0283808Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0284012Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0284217Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0284420Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0284623Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0284828Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0285032Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0285239Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0285445Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0285706Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0285910Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0286113Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0286312Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0286517Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0286723Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0286928Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0287132Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0287339Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0287548Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0287755Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0287952Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0288154Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0288878Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T03:00:28.0289591Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T03:00:28.0290285Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T03:00:28.0290969Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T03:00:28.0291224Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0291439Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0291848Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0292054Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0292265Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0292520Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0292731Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0292958Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0293164Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0293381Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0293592Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0293859Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0294072Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0294288Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0294501Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0294706Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0294920Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0295131Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0295349Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0295564Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0295774Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0295987Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0296199Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0296409Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0296619Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0297004Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0297384Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0297590Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0297800Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0298188Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0298399Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0298606Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0298818Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0299031Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0299244Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0299535Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0299757Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0299974Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0300188Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0300399Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0300603Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0300975Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0301181Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0301388Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0301595Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0301799Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0302241Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0302455Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0303208Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T03:00:28.0303931Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T03:00:28.0304653Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T03:00:28.0305507Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T03:00:28.0305721Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0305932Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0306142Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0306358Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0306567Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0306775Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0306981Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0307185Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0307276Z dist init r=3, world=4
2022-11-23T03:00:28.0307368Z dist init r=1, world=4
2022-11-23T03:00:28.0307459Z dist init r=0, world=4
2022-11-23T03:00:28.0307551Z dist init r=2, world=4
2022-11-23T03:00:28.0307638Z ok (41.697s)
2022-11-23T03:00:28.0307839Z   test_delayed_reduce_scatter_offload_true_no_shard (__main__.TestParityWithDDP)
2022-11-23T03:00:28.0308371Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.0308575Z   test_delayed_reduce_scatter_offload_true_none (__main__.TestParityWithDDP)
2022-11-23T03:00:28.0308859Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 123756
2022-11-23T03:00:28.0309060Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 123757
2022-11-23T03:00:28.0309434Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 123758
2022-11-23T03:00:28.0309639Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 123759
2022-11-23T03:00:28.0310003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:00:28.0310170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:00:28.0310594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:00:28.0310827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:00:28.0311183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:00:28.0311346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:00:28.0311705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:00:28.0311881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:00:28.0312376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:00:28.0312533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:00:28.0312880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:00:28.0313051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:00:28.0313385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:00:28.0313533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:00:28.0313886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:00:28.0314051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:00:28.0314274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:00:28.0314499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:00:28.0314724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:00:28.0314948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:00:28.0315321Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:00:28.0315694Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:00:28.0316052Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:00:28.0316415Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:00:28.0316625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:00:28.0316885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:00:28.0317099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:00:28.0317307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:00:28.0317522Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0317732Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0317939Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0318150Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0319129Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:00:28.0319277Z   warnings.warn(
2022-11-23T03:00:28.0320230Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:00:28.0320323Z   warnings.warn(
2022-11-23T03:00:28.0321267Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:00:28.0321363Z   warnings.warn(
2022-11-23T03:00:28.0322295Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:00:28.0322389Z   warnings.warn(
2022-11-23T03:00:28.0322502Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0322701Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0322828Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0323014Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0323149Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0323343Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0323433Z     self.run()
2022-11-23T03:00:28.0323545Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0323728Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0323857Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0324224Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0324349Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0324687Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0324871Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0325071Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0325214Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0325572Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0325682Z     getattr(self, test_name)()
2022-11-23T03:00:28.0325885Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0325970Z     self.run()
2022-11-23T03:00:28.0326086Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0326440Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0326526Z     fn()
2022-11-23T03:00:28.0326717Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0326853Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0327051Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0327226Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0327745Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0327852Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0328171Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0328288Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0328469Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0328602Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0328938Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0329042Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0329382Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0329493Z     getattr(self, test_name)()
2022-11-23T03:00:28.0329687Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0329775Z     self.run()
2022-11-23T03:00:28.0330007Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0330104Z     self.run_subtests(
2022-11-23T03:00:28.0330443Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0330519Z     fn()
2022-11-23T03:00:28.0330871Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0331009Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0331359Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0331512Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0331872Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0331984Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0332302Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0332423Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0332779Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0332919Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0333264Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0333376Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0333925Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0334041Z     getattr(self, test_name)()
2022-11-23T03:00:28.0334394Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0334500Z     output = model(*input)
2022-11-23T03:00:28.0334734Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0334830Z     self.run_subtests(
2022-11-23T03:00:28.0335342Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0335426Z     fn()
2022-11-23T03:00:28.0335740Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0335869Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0336207Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0336370Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0337014Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0337136Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0337512Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0337674Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0338023Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0338165Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0338512Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0338621Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0338979Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0339090Z     _lazy_init(state, module)
2022-11-23T03:00:28.0339455Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0339561Z     output = model(*input)
2022-11-23T03:00:28.0339802Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0339903Z     self.run_subtests(
2022-11-23T03:00:28.0340247Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0340373Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0340689Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0340819Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0341162Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0341315Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0341646Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0341757Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0342277Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0342430Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0342787Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0343044Z     p_assert(
2022-11-23T03:00:28.0343475Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0343626Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0343989Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0344097Z     _lazy_init(state, module)
2022-11-23T03:00:28.0344421Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0344530Z     traceback.print_stack()
2022-11-23T03:00:28.0344895Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0345001Z     output = model(*input)
2022-11-23T03:00:28.0345344Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0345476Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0345791Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0345918Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0346315Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0346421Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0346786Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0346947Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0347467Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0347554Z     p_assert(
2022-11-23T03:00:28.0347895Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0347999Z     _lazy_init(state, module)
2022-11-23T03:00:28.0348317Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0348422Z     traceback.print_stack()
2022-11-23T03:00:28.0348757Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0348883Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0349199Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0349306Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0349658Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0349746Z     p_assert(
2022-11-23T03:00:28.0350054Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0350157Z     traceback.print_stack()
2022-11-23T03:00:28.0350268Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0350464Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0350592Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0350775Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0350909Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0351103Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0351350Z     self.run()
2022-11-23T03:00:28.0351543Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0351676Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0352005Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0352125Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0352472Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0352628Z     getattr(self, test_name)()
2022-11-23T03:00:28.0352987Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0353071Z     fn()
2022-11-23T03:00:28.0353425Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0353534Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0353875Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0353986Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0354378Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0354475Z     self.run_subtests(
2022-11-23T03:00:28.0354805Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0354950Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0355294Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0355481Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0355835Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0355937Z     output = model(*input)
2022-11-23T03:00:28.0356241Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0356368Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0356887Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0357045Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0357406Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0357515Z     _lazy_init(state, module)
2022-11-23T03:00:28.0357856Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0357985Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0358313Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0358423Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0358789Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0358872Z     p_assert(
2022-11-23T03:00:28.0359196Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0359309Z     traceback.print_stack()
2022-11-23T03:00:28.0359693Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0359911Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0360132Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0360344Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0360457Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0360646Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0360771Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0360952Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0361087Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0361284Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0361372Z     self.run()
2022-11-23T03:00:28.0361774Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0361909Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0362247Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0362367Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0362719Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0362829Z     getattr(self, test_name)()
2022-11-23T03:00:28.0363178Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0363266Z     fn()
2022-11-23T03:00:28.0363620Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0363724Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0364069Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0364180Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0364469Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0364569Z     self.run_subtests(
2022-11-23T03:00:28.0364914Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0365065Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0365415Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0365551Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0365917Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0366024Z     output = model(*input)
2022-11-23T03:00:28.0366493Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0366618Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0366971Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0367128Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0367472Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0367570Z     _lazy_init(state, module)
2022-11-23T03:00:28.0367898Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0368024Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0368341Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0368452Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0368565Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0369107Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0369197Z     p_assert(
2022-11-23T03:00:28.0369516Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0369630Z     traceback.print_stack()
2022-11-23T03:00:28.0369829Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0369959Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0370149Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0370287Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0370488Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0370578Z     self.run()
2022-11-23T03:00:28.0370809Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0370953Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0371286Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0371405Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0371913Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0372021Z     getattr(self, test_name)()
2022-11-23T03:00:28.0372355Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0372430Z     fn()
2022-11-23T03:00:28.0372771Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0372879Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0373212Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0373370Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0373603Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0373701Z     self.run_subtests(
2022-11-23T03:00:28.0373815Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0374143Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0374289Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0374630Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0374765Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0374955Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0375083Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0375196Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0375550Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0375648Z     output = model(*input)
2022-11-23T03:00:28.0376010Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0376147Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0376462Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0376589Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0377007Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0377144Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0377350Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0377440Z     self.run()
2022-11-23T03:00:28.0377819Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0377987Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0378176Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0378315Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0378504Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0378638Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0378988Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0379096Z     _lazy_init(state, module)
2022-11-23T03:00:28.0379297Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0379386Z     self.run()
2022-11-23T03:00:28.0379811Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0379918Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0380262Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0380392Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0380577Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0380712Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0381060Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0381169Z     getattr(self, test_name)()
2022-11-23T03:00:28.0381553Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0381665Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0381992Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0382177Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0382525Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0382610Z     fn()
2022-11-23T03:00:28.0382976Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0383065Z     p_assert(
2022-11-23T03:00:28.0383414Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0383523Z     getattr(self, test_name)()
2022-11-23T03:00:28.0384022Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0384127Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0384437Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0384549Z     traceback.print_stack()
2022-11-23T03:00:28.0384890Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0384971Z     fn()
2022-11-23T03:00:28.0385302Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0385413Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0385753Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0385859Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0386085Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0386183Z     self.run_subtests(
2022-11-23T03:00:28.0386520Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0386627Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0386962Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0387108Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0387341Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0387438Z     self.run_subtests(
2022-11-23T03:00:28.0387771Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0387906Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0388235Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0388379Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0388786Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0388904Z     output = model(*input)
2022-11-23T03:00:28.0389249Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0389384Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0389681Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0389807Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0390159Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0390261Z     output = model(*input)
2022-11-23T03:00:28.0390611Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0390774Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0391081Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0391257Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0391597Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0391704Z     _lazy_init(state, module)
2022-11-23T03:00:28.0392236Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0392440Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0392785Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0392927Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0393273Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0393381Z     _lazy_init(state, module)
2022-11-23T03:00:28.0393707Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0393821Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0394159Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0394289Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0394661Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0394752Z     p_assert(
2022-11-23T03:00:28.0395077Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0395188Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0395511Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0395627Z     traceback.print_stack()
2022-11-23T03:00:28.0395998Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0396086Z     p_assert(
2022-11-23T03:00:28.0396408Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0396523Z     traceback.print_stack()
2022-11-23T03:00:28.0396748Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0396971Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0397181Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0397405Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0397523Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0397771Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0397911Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0398103Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0398241Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0398442Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0398528Z     self.run()
2022-11-23T03:00:28.0398719Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0398857Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0399194Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0399316Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0399672Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0399782Z     getattr(self, test_name)()
2022-11-23T03:00:28.0400124Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0400259Z     fn()
2022-11-23T03:00:28.0400618Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0400729Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0401227Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0401336Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0401567Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0401665Z     self.run_subtests(
2022-11-23T03:00:28.0401988Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0402137Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0402656Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0402798Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0403159Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0403265Z     output = model(*input)
2022-11-23T03:00:28.0403580Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0403708Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0404065Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0404227Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0404589Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0404698Z     _lazy_init(state, module)
2022-11-23T03:00:28.0405037Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0405327Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0405649Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0405756Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0406102Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0406189Z     p_assert(
2022-11-23T03:00:28.0406503Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0406613Z     traceback.print_stack()
2022-11-23T03:00:28.0406726Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0407002Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0407140Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0407323Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0407451Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0407646Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0407735Z     self.run()
2022-11-23T03:00:28.0407916Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0408045Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0408368Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0408484Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0408823Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0408925Z     getattr(self, test_name)()
2022-11-23T03:00:28.0409084Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0409422Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0409676Z     fn()
2022-11-23T03:00:28.0410029Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0410139Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0410335Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0410461Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0410868Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0410980Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0411174Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0411314Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0411554Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0411654Z     self.run_subtests(
2022-11-23T03:00:28.0411857Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0411941Z     self.run()
2022-11-23T03:00:28.0412283Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0412581Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0412766Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0412894Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0413233Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0413369Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0413690Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0413801Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0414152Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0414254Z     output = model(*input)
2022-11-23T03:00:28.0414593Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0414698Z     getattr(self, test_name)()
2022-11-23T03:00:28.0415002Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0415125Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0415505Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0415589Z     fn()
2022-11-23T03:00:28.0415945Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0416281Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0416634Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0416936Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0417301Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0417408Z     _lazy_init(state, module)
2022-11-23T03:00:28.0417754Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0417859Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0417975Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0418320Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0418532Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0418773Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0418872Z     self.run_subtests(
2022-11-23T03:00:28.0419352Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0419459Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0419647Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0419772Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0420110Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0420254Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0420611Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0420702Z     p_assert(
2022-11-23T03:00:28.0420887Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0421015Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0421360Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0421496Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0421811Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0421921Z     traceback.print_stack()
2022-11-23T03:00:28.0422114Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0422202Z     self.run()
2022-11-23T03:00:28.0422559Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0422657Z     output = model(*input)
2022-11-23T03:00:28.0422844Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0422975Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0423281Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0423405Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0423721Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0423837Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0424188Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0424512Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0424925Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0425049Z     getattr(self, test_name)()
2022-11-23T03:00:28.0425411Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0425520Z     _lazy_init(state, module)
2022-11-23T03:00:28.0425868Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0425957Z     fn()
2022-11-23T03:00:28.0426297Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0426421Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0426777Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0426887Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0427218Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0427330Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0427734Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0428002Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0428357Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0428437Z     p_assert(
2022-11-23T03:00:28.0428668Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0428766Z     self.run_subtests(
2022-11-23T03:00:28.0429256Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0429371Z     traceback.print_stack()
2022-11-23T03:00:28.0429714Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0429866Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0430221Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0430355Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0430720Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0430826Z     output = model(*input)
2022-11-23T03:00:28.0431139Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0431267Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0431632Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0431797Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0432308Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0432410Z     _lazy_init(state, module)
2022-11-23T03:00:28.0432738Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0432863Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0433179Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0433287Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0433642Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0433727Z     p_assert(
2022-11-23T03:00:28.0434039Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0434142Z     traceback.print_stack()
2022-11-23T03:00:28.0434407Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0434635Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0434850Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0435062Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0435175Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0435537Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0435669Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0435852Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0435990Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0436189Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0436282Z     self.run()
2022-11-23T03:00:28.0436472Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0436655Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0436993Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0437109Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0437461Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0437571Z     getattr(self, test_name)()
2022-11-23T03:00:28.0437917Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0438002Z     fn()
2022-11-23T03:00:28.0438355Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0438465Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0438814Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0438923Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0439166Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0439266Z     self.run_subtests(
2022-11-23T03:00:28.0439608Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0439758Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0440108Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0440247Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0440610Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0440716Z     output = model(*input)
2022-11-23T03:00:28.0441033Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0441166Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0441532Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0441696Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0442049Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0442158Z     _lazy_init(state, module)
2022-11-23T03:00:28.0442656Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0442784Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0443144Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0443435Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0443813Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0443918Z     p_assert(
2022-11-23T03:00:28.0444227Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0444340Z     traceback.print_stack()
2022-11-23T03:00:28.0444456Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0444646Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0444775Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0444965Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0445105Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0445306Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0445401Z     self.run()
2022-11-23T03:00:28.0445591Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0445772Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0446099Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0446221Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0446571Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0446682Z     getattr(self, test_name)()
2022-11-23T03:00:28.0447027Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0447112Z     fn()
2022-11-23T03:00:28.0447467Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0447733Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0448060Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0448173Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0448409Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0448506Z     self.run_subtests(
2022-11-23T03:00:28.0448836Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0448980Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0449318Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0449453Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0449800Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0449909Z     output = model(*input)
2022-11-23T03:00:28.0450021Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0450332Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0450455Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0450806Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0450963Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0451154Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0451444Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0451801Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0451909Z     _lazy_init(state, module)
2022-11-23T03:00:28.0452084Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0452283Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0452427Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0452776Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0452899Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0453097Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0453229Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0453432Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0453524Z     self.run()
2022-11-23T03:00:28.0453854Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0453966Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0454159Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0454292Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0454536Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0454670Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0455041Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0455132Z     p_assert(
2022-11-23T03:00:28.0455485Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0455574Z     self.run()
2022-11-23T03:00:28.0455884Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0455999Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0456314Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0456428Z     traceback.print_stack()
2022-11-23T03:00:28.0456615Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0456933Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0457462Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0457572Z     getattr(self, test_name)()
2022-11-23T03:00:28.0457894Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0458017Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0458366Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0458454Z     fn()
2022-11-23T03:00:28.0458803Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0458914Z     getattr(self, test_name)()
2022-11-23T03:00:28.0459271Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0459383Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0459722Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0459806Z     fn()
2022-11-23T03:00:28.0460293Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0460401Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0460745Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0460850Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0461081Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0461179Z     self.run_subtests(
2022-11-23T03:00:28.0461578Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0461698Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0462228Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0462359Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0462597Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0462698Z     self.run_subtests(
2022-11-23T03:00:28.0463050Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0463190Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0463521Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0463675Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0464044Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0464212Z     output = model(*input)
2022-11-23T03:00:28.0464570Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0464710Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0465026Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0465157Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0465516Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0465626Z     output = model(*input)
2022-11-23T03:00:28.0465992Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0466156Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0466629Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0466754Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0467098Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0467202Z     _lazy_init(state, module)
2022-11-23T03:00:28.0467739Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0467892Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0468230Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0468363Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0468721Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0468832Z     _lazy_init(state, module)
2022-11-23T03:00:28.0469160Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0469273Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0469608Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0469741Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0470111Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0470200Z     p_assert(
2022-11-23T03:00:28.0470680Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0470789Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0471152Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0471273Z     traceback.print_stack()
2022-11-23T03:00:28.0471623Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0471711Z     p_assert(
2022-11-23T03:00:28.0472026Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0472135Z     traceback.print_stack()
2022-11-23T03:00:28.0472354Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0472572Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0472786Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0473000Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0473110Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0473307Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0473502Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0473685Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0473819Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0474014Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0474102Z     self.run()
2022-11-23T03:00:28.0474286Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0474412Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0474736Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0474852Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0475196Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0475305Z     getattr(self, test_name)()
2022-11-23T03:00:28.0475643Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0475726Z     fn()
2022-11-23T03:00:28.0476232Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0476346Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0476688Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0476799Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0477041Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0477142Z     self.run_subtests(
2022-11-23T03:00:28.0477488Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0477639Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0477987Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0478129Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0478494Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0478601Z     output = model(*input)
2022-11-23T03:00:28.0478913Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0479042Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0479409Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0479621Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0479739Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0480102Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0480212Z     _lazy_init(state, module)
2022-11-23T03:00:28.0480554Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0480685Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0480883Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0481012Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0481338Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0481443Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0481632Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0481774Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0482141Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0482280Z     p_assert(
2022-11-23T03:00:28.0482481Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0482571Z     self.run()
2022-11-23T03:00:28.0482900Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0483007Z     traceback.print_stack()
2022-11-23T03:00:28.0483199Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0483332Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0483660Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0483779Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0484283Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0484396Z     getattr(self, test_name)()
2022-11-23T03:00:28.0484732Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0484807Z     fn()
2022-11-23T03:00:28.0485150Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0485256Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0485589Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0485695Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0485930Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0486028Z     self.run_subtests(
2022-11-23T03:00:28.0486363Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0486506Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0486848Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0486981Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0487334Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0487437Z     output = model(*input)
2022-11-23T03:00:28.0487740Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0487863Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0488215Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0488415Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0488767Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0488874Z     _lazy_init(state, module)
2022-11-23T03:00:28.0489201Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0489328Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0489643Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0489750Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0490103Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0490185Z     p_assert(
2022-11-23T03:00:28.0490499Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0490612Z     traceback.print_stack()
2022-11-23T03:00:28.0490726Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0490968Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0491094Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0491277Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0491405Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0491599Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0491686Z     self.run()
2022-11-23T03:00:28.0491869Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0491997Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0492519Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0492657Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0493020Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0493130Z     getattr(self, test_name)()
2022-11-23T03:00:28.0493480Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0493565Z     fn()
2022-11-23T03:00:28.0493919Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0494030Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0494379Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0494491Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0494731Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0494827Z     self.run_subtests(
2022-11-23T03:00:28.0495171Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0495326Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0495681Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0495822Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0495938Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0496303Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0496410Z     output = model(*input)
2022-11-23T03:00:28.0496895Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0497042Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0497241Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0497441Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0497825Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0497991Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0498182Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0498323Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0498676Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0498786Z     _lazy_init(state, module)
2022-11-23T03:00:28.0498987Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0499078Z     self.run()
2022-11-23T03:00:28.0499420Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0499562Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0499756Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0499950Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0500284Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0500398Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0500724Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0500844Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0501211Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0501460Z     p_assert(
2022-11-23T03:00:28.0501799Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0501903Z     getattr(self, test_name)()
2022-11-23T03:00:28.0502217Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0502330Z     traceback.print_stack()
2022-11-23T03:00:28.0502834Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0502942Z     fn()
2022-11-23T03:00:28.0503275Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0503385Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0503734Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0503839Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0504082Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0504184Z     self.run_subtests(
2022-11-23T03:00:28.0504530Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0504683Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0505035Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0505174Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0505691Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0505789Z     output = model(*input)
2022-11-23T03:00:28.0506095Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0506218Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0506572Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0506778Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0507132Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0507277Z     _lazy_init(state, module)
2022-11-23T03:00:28.0507671Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0507795Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0508161Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0508316Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0508750Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0508884Z     p_assert(
2022-11-23T03:00:28.0509246Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0509407Z     traceback.print_stack()
2022-11-23T03:00:28.0509625Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0510209Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0510484Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0510814Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0510979Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0511228Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0511353Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0511587Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0511781Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0512034Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0512211Z     self.run()
2022-11-23T03:00:28.0512458Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0512640Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0512989Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0513316Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0513480Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0513876Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0514027Z     getattr(self, test_name)()
2022-11-23T03:00:28.0514263Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0514472Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0514866Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0514993Z     fn()
2022-11-23T03:00:28.0515174Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0515352Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0515937Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0516096Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0516341Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0516480Z     self.run()
2022-11-23T03:00:28.0516909Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0517019Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0517257Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0517531Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0517837Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0517988Z     self.run_subtests(
2022-11-23T03:00:28.0518152Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0518692Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0518856Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0518962Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0519381Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0519574Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0519822Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0519993Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0520384Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0520585Z     getattr(self, test_name)()
2022-11-23T03:00:28.0520981Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0521112Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0521349Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0521561Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0521829Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0522008Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0522398Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0522528Z     fn()
2022-11-23T03:00:28.0522879Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0523035Z     output = model(*input)
2022-11-23T03:00:28.0523266Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0523454Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0523841Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0524014Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0524255Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0524388Z     self.run()
2022-11-23T03:00:28.0524900Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0525064Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0525312Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0525463Z     self.run()
2022-11-23T03:00:28.0525880Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0526126Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0526403Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0526585Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0526935Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0527096Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0527334Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0527613Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0528031Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0528400Z     _lazy_init(state, module)
2022-11-23T03:00:28.0528815Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0528985Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0529213Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0529358Z     self.run_subtests(
2022-11-23T03:00:28.0529731Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0529896Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0530275Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0530448Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0530866Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0531060Z     getattr(self, test_name)()
2022-11-23T03:00:28.0531392Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0531891Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0532301Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0532460Z     getattr(self, test_name)()
2022-11-23T03:00:28.0532837Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0532996Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0533410Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0533541Z     fn()
2022-11-23T03:00:28.0533883Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0534064Z     fn()
2022-11-23T03:00:28.0534625Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0534811Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0535218Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0535354Z     p_assert(
2022-11-23T03:00:28.0535906Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0536103Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0536426Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0536597Z     traceback.print_stack()
2022-11-23T03:00:28.0537250Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0537415Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0537838Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0537996Z     output = model(*input)
2022-11-23T03:00:28.0538392Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0538499Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0538893Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0539063Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0539428Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0539637Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0539929Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0540156Z     self.run_subtests(
2022-11-23T03:00:28.0540449Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0540548Z     self.run_subtests(
2022-11-23T03:00:28.0540969Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0541231Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0541620Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0541815Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0542257Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0542415Z     _lazy_init(state, module)
2022-11-23T03:00:28.0542808Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0543005Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0543434Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0543623Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0544013Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0544193Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0544593Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0544815Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0545233Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0545397Z     output = model(*input)
2022-11-23T03:00:28.0545727Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0545891Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0546341Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0546494Z     output = model(*input)
2022-11-23T03:00:28.0546856Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0547030Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0547477Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0547624Z     p_assert(
2022-11-23T03:00:28.0548091Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0548266Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0548670Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0548877Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0549236Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0549390Z     traceback.print_stack()
2022-11-23T03:00:28.0549791Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0550039Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0550582Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0550715Z     _lazy_init(state, module)
2022-11-23T03:00:28.0551116Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0551351Z     _lazy_init(state, module)
2022-11-23T03:00:28.0551746Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0551929Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0552313Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0552499Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0552821Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0553014Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0553387Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0553548Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0553964Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0554104Z     p_assert(
2022-11-23T03:00:28.0554662Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0554852Z     p_assert(
2022-11-23T03:00:28.0555167Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0555323Z     traceback.print_stack()
2022-11-23T03:00:28.0555718Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0555880Z     traceback.print_stack()
2022-11-23T03:00:28.0556179Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0556443Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0556712Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0556979Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0557092Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0557332Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0557669Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0557937Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0558125Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0558376Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0558525Z     self.run()
2022-11-23T03:00:28.0558713Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0558892Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0559272Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0559446Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0559846Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0569745Z     getattr(self, test_name)()
2022-11-23T03:00:28.0570181Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0570270Z     fn()
2022-11-23T03:00:28.0570629Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0570733Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0571074Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0571184Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0571420Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0571518Z     self.run_subtests(
2022-11-23T03:00:28.0571951Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0572114Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0572462Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0572592Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0573118Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0573231Z     output = model(*input)
2022-11-23T03:00:28.0573547Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0573677Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0574044Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0574213Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0574632Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0574742Z     _lazy_init(state, module)
2022-11-23T03:00:28.0575075Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0575206Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0575536Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0575648Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0576168Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0576258Z     p_assert(
2022-11-23T03:00:28.0577041Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0577162Z     traceback.print_stack()
2022-11-23T03:00:28.0577281Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0577492Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0577617Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0577808Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0577947Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0578156Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0578248Z     self.run()
2022-11-23T03:00:28.0578430Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0578563Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0578903Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0579031Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0579384Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0579498Z     getattr(self, test_name)()
2022-11-23T03:00:28.0579849Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0579935Z     fn()
2022-11-23T03:00:28.0580288Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0580400Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0580751Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0580864Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0581106Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0581294Z     self.run_subtests(
2022-11-23T03:00:28.0581654Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0581812Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0582159Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0582299Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0582416Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0582782Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0582891Z     output = model(*input)
2022-11-23T03:00:28.0583205Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0583335Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0583533Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0583666Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0584145Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0584461Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0584650Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0584785Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0585130Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0585237Z     _lazy_init(state, module)
2022-11-23T03:00:28.0585428Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0585517Z     self.run()
2022-11-23T03:00:28.0585628Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0585962Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0586092Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0586278Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0586408Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0586728Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0586830Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0587023Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0587148Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0587464Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0587581Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0587943Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0588032Z     p_assert(
2022-11-23T03:00:28.0588216Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0588346Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0588689Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0588795Z     getattr(self, test_name)()
2022-11-23T03:00:28.0589111Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0589223Z     traceback.print_stack()
2022-11-23T03:00:28.0589417Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0589504Z     self.run()
2022-11-23T03:00:28.0589837Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0589965Z     fn()
2022-11-23T03:00:28.0590160Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0590292Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0590641Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0590747Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0591059Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0591174Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0591505Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0591614Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0591948Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0592054Z     getattr(self, test_name)()
2022-11-23T03:00:28.0592291Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0592455Z     self.run_subtests(
2022-11-23T03:00:28.0593045Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0593130Z     fn()
2022-11-23T03:00:28.0593465Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0593616Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0593970Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0594082Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0594427Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0594572Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0594919Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0595036Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0595394Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0595499Z     output = model(*input)
2022-11-23T03:00:28.0595743Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0595844Z     self.run_subtests(
2022-11-23T03:00:28.0596160Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0596292Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0596633Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0596792Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0597154Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0597323Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0597832Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0597968Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0598311Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0598417Z     _lazy_init(state, module)
2022-11-23T03:00:28.0598941Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0599049Z     output = model(*input)
2022-11-23T03:00:28.0599434Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0599573Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0599894Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0600022Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0600349Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0600461Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0600822Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0600986Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0601348Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0601440Z     p_assert(
2022-11-23T03:00:28.0601954Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0602109Z     _lazy_init(state, module)
2022-11-23T03:00:28.0602427Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0602538Z     traceback.print_stack()
2022-11-23T03:00:28.0602864Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0602994Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0603487Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0603600Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0603966Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0604055Z     p_assert(
2022-11-23T03:00:28.0604384Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0604502Z     traceback.print_stack()
2022-11-23T03:00:28.0604732Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0604956Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0605168Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0605392Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0605509Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0605625Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0605825Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0606113Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0606308Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0606443Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0606631Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0606757Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0606953Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0607040Z     self.run()
2022-11-23T03:00:28.0607223Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0607358Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0607542Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0607665Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0607859Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0607947Z     self.run()
2022-11-23T03:00:28.0608319Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0608441Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0608629Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0608757Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0609098Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0609199Z     getattr(self, test_name)()
2022-11-23T03:00:28.0609509Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0609624Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0609962Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0610044Z     fn()
2022-11-23T03:00:28.0610558Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0610726Z     getattr(self, test_name)()
2022-11-23T03:00:28.0611143Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0611247Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0611594Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0611680Z     fn()
2022-11-23T03:00:28.0612017Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0612129Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0612483Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0612592Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0612837Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0612932Z     self.run_subtests(
2022-11-23T03:00:28.0613427Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0613538Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0613868Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0614012Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0614242Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0614338Z     self.run_subtests(
2022-11-23T03:00:28.0614677Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0614806Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0615137Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0615279Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0615633Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0615736Z     output = model(*input)
2022-11-23T03:00:28.0616074Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0616209Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0616509Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0616629Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0617194Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0617296Z     output = model(*input)
2022-11-23T03:00:28.0617719Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0617893Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0618196Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0618319Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0618661Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0618760Z     _lazy_init(state, module)
2022-11-23T03:00:28.0619108Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0619264Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0619593Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0619725Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0620069Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0620235Z     _lazy_init(state, module)
2022-11-23T03:00:28.0620563Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0620665Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0620992Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0621117Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0621470Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0621560Z     p_assert(
2022-11-23T03:00:28.0621875Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0621986Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0622299Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0622407Z     traceback.print_stack()
2022-11-23T03:00:28.0622760Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0622847Z     p_assert(
2022-11-23T03:00:28.0623158Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0623269Z     traceback.print_stack()
2022-11-23T03:00:28.0623381Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0623491Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0623677Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0623803Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0623991Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0624126Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0624321Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0624444Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0624639Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0624726Z     self.run()
2022-11-23T03:00:28.0624901Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0625208Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0625401Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0625535Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0625737Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0625827Z     self.run()
2022-11-23T03:00:28.0626205Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0626338Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0626523Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0626656Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0627010Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0627120Z     getattr(self, test_name)()
2022-11-23T03:00:28.0627447Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0627566Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0627913Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0627991Z     fn()
2022-11-23T03:00:28.0628340Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0628451Z     getattr(self, test_name)()
2022-11-23T03:00:28.0629002Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0629108Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0629441Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0629523Z     fn()
2022-11-23T03:00:28.0629849Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0629951Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0630291Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0630396Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0630632Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0630730Z     self.run_subtests(
2022-11-23T03:00:28.0631237Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0631349Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0631692Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0631836Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0632075Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0632176Z     self.run_subtests(
2022-11-23T03:00:28.0632526Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0632667Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0633009Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0633161Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0633529Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0633630Z     output = model(*input)
2022-11-23T03:00:28.0633981Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0634273Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0634577Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0634702Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0635051Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0635198Z     output = model(*input)
2022-11-23T03:00:28.0635560Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0635717Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0636023Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0636321Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0636692Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0636785Z     _lazy_init(state, module)
2022-11-23T03:00:28.0637144Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0637308Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0637653Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0637784Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0638185Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0638294Z     _lazy_init(state, module)
2022-11-23T03:00:28.0638620Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0638732Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0639072Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0639204Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0639571Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0639661Z     p_assert(
2022-11-23T03:00:28.0639982Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0640094Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0640422Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0640536Z     traceback.print_stack()
2022-11-23T03:00:28.0640898Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0640988Z     p_assert(
2022-11-23T03:00:28.0641312Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0641418Z     traceback.print_stack()
2022-11-23T03:00:28.0641642Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0641865Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0642090Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0642312Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0642433Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0642631Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0642762Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0642947Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0643084Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0643437Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0643525Z     self.run()
2022-11-23T03:00:28.0643882Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0644016Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0644398Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0644525Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0644878Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0644989Z     getattr(self, test_name)()
2022-11-23T03:00:28.0645338Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0645424Z     fn()
2022-11-23T03:00:28.0645774Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0645885Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0646226Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0646337Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0646579Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0646681Z     self.run_subtests(
2022-11-23T03:00:28.0647025Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0647228Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0647581Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0647721Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0648083Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0648343Z     output = model(*input)
2022-11-23T03:00:28.0648640Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0648765Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0649119Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0649276Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0649621Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0649726Z     _lazy_init(state, module)
2022-11-23T03:00:28.0650051Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0650177Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0650485Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0650593Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0650949Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0651037Z     p_assert(
2022-11-23T03:00:28.0651353Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0651466Z     traceback.print_stack()
2022-11-23T03:00:28.0651746Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0651939Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0652072Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0652262Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0652401Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0652603Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0652694Z     self.run()
2022-11-23T03:00:28.0652881Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0653016Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0653390Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0653517Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0653872Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0653982Z     getattr(self, test_name)()
2022-11-23T03:00:28.0654329Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0654415Z     fn()
2022-11-23T03:00:28.0654916Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0655024Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0655350Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0655458Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0655693Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0655792Z     self.run_subtests(
2022-11-23T03:00:28.0656193Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0656337Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0656677Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0657185Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0657548Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0657658Z     output = model(*input)
2022-11-23T03:00:28.0657971Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0658101Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0658219Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0658590Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0658757Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0659119Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0659221Z     _lazy_init(state, module)
2022-11-23T03:00:28.0659419Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0659549Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0659890Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0660020Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0660212Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0660353Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0660834Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0660942Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0661135Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0661222Z     self.run()
2022-11-23T03:00:28.0661585Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0661674Z     p_assert(
2022-11-23T03:00:28.0661785Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0661970Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0662093Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0662413Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0662521Z     traceback.print_stack()
2022-11-23T03:00:28.0663086Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0663221Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0663419Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0663549Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0663907Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0664011Z     getattr(self, test_name)()
2022-11-23T03:00:28.0664198Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0664337Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0664690Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0664774Z     fn()
2022-11-23T03:00:28.0664980Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0665070Z     self.run()
2022-11-23T03:00:28.0665496Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0665601Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0665790Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0665927Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0666277Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0666388Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0666712Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0666832Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0667071Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0667329Z     self.run_subtests(
2022-11-23T03:00:28.0667671Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0667782Z     getattr(self, test_name)()
2022-11-23T03:00:28.0668110Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0668255Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0668588Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0668670Z     fn()
2022-11-23T03:00:28.0669001Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0669137Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0669667Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0669778Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0670145Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0670252Z     output = model(*input)
2022-11-23T03:00:28.0670599Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0670710Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0671016Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0671145Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0671384Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0671484Z     self.run_subtests(
2022-11-23T03:00:28.0671912Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0672238Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0672577Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0672722Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0673057Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0673163Z     _lazy_init(state, module)
2022-11-23T03:00:28.0673501Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0673636Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0673965Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0674091Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0674445Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0674596Z     output = model(*input)
2022-11-23T03:00:28.0674916Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0675018Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0675324Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0675450Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0675803Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0675898Z     p_assert(
2022-11-23T03:00:28.0676248Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0676409Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0676722Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0677006Z     traceback.print_stack()
2022-11-23T03:00:28.0677360Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0677468Z     _lazy_init(state, module)
2022-11-23T03:00:28.0677806Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0677937Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0678263Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0678375Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0678740Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0678828Z     p_assert(
2022-11-23T03:00:28.0679152Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0679270Z     traceback.print_stack()
2022-11-23T03:00:28.0679502Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0679726Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0679943Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0680161Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0680272Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0680475Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0680605Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0680842Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0680989Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0681193Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0681296Z     self.run()
2022-11-23T03:00:28.0681520Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0681649Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0681987Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0682106Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0682458Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0682569Z     getattr(self, test_name)()
2022-11-23T03:00:28.0682916Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0683000Z     fn()
2022-11-23T03:00:28.0683360Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0683518Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0683864Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0683975Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0684215Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0684315Z     self.run_subtests(
2022-11-23T03:00:28.0684655Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0684952Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0685461Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0685601Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0685966Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0686076Z     output = model(*input)
2022-11-23T03:00:28.0686389Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0686518Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0686882Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0687047Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0687404Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0687507Z     _lazy_init(state, module)
2022-11-23T03:00:28.0687848Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0687979Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0688309Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0688420Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0688930Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0689016Z     p_assert(
2022-11-23T03:00:28.0689328Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0689432Z     traceback.print_stack()
2022-11-23T03:00:28.0689545Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0689736Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0689862Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0690093Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0690235Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0690433Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0690515Z     self.run()
2022-11-23T03:00:28.0690699Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0690827Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0691146Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0691262Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0691601Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0691707Z     getattr(self, test_name)()
2022-11-23T03:00:28.0692040Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0692120Z     fn()
2022-11-23T03:00:28.0692463Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0692663Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0693007Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0693297Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0693551Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0693644Z     self.run_subtests(
2022-11-23T03:00:28.0693985Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0694129Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0694477Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0694621Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0694989Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0695099Z     output = model(*input)
2022-11-23T03:00:28.0695412Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0695540Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0695906Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0696064Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0696420Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0696528Z     _lazy_init(state, module)
2022-11-23T03:00:28.0697065Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0697202Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0697541Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0697673Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0698021Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0698104Z     p_assert(
2022-11-23T03:00:28.0698222Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0698544Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0698658Z     traceback.print_stack()
2022-11-23T03:00:28.0698856Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0698986Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0699252Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0699401Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0699602Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0699693Z     self.run()
2022-11-23T03:00:28.0699884Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0700017Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0700349Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0700470Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0700822Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0700927Z     getattr(self, test_name)()
2022-11-23T03:00:28.0701271Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0701359Z     fn()
2022-11-23T03:00:28.0701712Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0702043Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0702155Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0702491Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0702598Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0702823Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0702919Z     self.run_subtests(
2022-11-23T03:00:28.0703110Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0703235Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0703744Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0703893Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0704085Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0704225Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0704572Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0704713Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0704914Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0705005Z     self.run()
2022-11-23T03:00:28.0705369Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0705475Z     output = model(*input)
2022-11-23T03:00:28.0705669Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0705806Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0706120Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0706252Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0706732Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0706849Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0707199Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0707356Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0707693Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0707802Z     getattr(self, test_name)()
2022-11-23T03:00:28.0708185Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0708294Z     _lazy_init(state, module)
2022-11-23T03:00:28.0708636Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0708718Z     fn()
2022-11-23T03:00:28.0709044Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0709170Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0709508Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0709613Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0709920Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0710027Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0710360Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0710467Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0711122Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0711212Z     p_assert(
2022-11-23T03:00:28.0711451Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0711546Z     self.run_subtests(
2022-11-23T03:00:28.0711869Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0711982Z     traceback.print_stack()
2022-11-23T03:00:28.0712322Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0712471Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0712821Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0712960Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0713327Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0713587Z     output = model(*input)
2022-11-23T03:00:28.0713884Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0714008Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0714362Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0714519Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0714861Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0714965Z     _lazy_init(state, module)
2022-11-23T03:00:28.0715293Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0715422Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0715731Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0715839Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0716192Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0716278Z     p_assert(
2022-11-23T03:00:28.0716765Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0716880Z     traceback.print_stack()
2022-11-23T03:00:28.0717108Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0717326Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0717597Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0717831Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0717948Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0718145Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0718274Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0718465Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0718603Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0718799Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0718891Z     self.run()
2022-11-23T03:00:28.0719081Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0719214Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0719705Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0719880Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0720221Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0720327Z     getattr(self, test_name)()
2022-11-23T03:00:28.0720657Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0720739Z     fn()
2022-11-23T03:00:28.0721080Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0721187Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0721514Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0721626Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0721866Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0721966Z     self.run_subtests(
2022-11-23T03:00:28.0722291Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0722438Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0722780Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0722916Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0723267Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0723371Z     output = model(*input)
2022-11-23T03:00:28.0723673Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0723798Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0724145Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0724306Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0724653Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0724756Z     _lazy_init(state, module)
2022-11-23T03:00:28.0725082Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0725208Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0725695Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0725810Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0726170Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0726345Z     p_assert(
2022-11-23T03:00:28.0726685Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0726801Z     traceback.print_stack()
2022-11-23T03:00:28.0726917Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0727117Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0727248Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0727439Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0727572Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0727774Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0727864Z     self.run()
2022-11-23T03:00:28.0728054Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0728185Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0728523Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0728694Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0729189Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0729297Z     getattr(self, test_name)()
2022-11-23T03:00:28.0729818Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0729903Z     fn()
2022-11-23T03:00:28.0730257Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0730367Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0730714Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0730825Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0731064Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0731169Z     self.run_subtests(
2022-11-23T03:00:28.0731512Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0731662Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0732011Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0732152Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0732671Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0732775Z     output = model(*input)
2022-11-23T03:00:28.0733072Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0733196Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0733548Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0733709Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0734050Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0734154Z     _lazy_init(state, module)
2022-11-23T03:00:28.0734481Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0734607Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0734916Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0735023Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0735376Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0735510Z     p_assert(
2022-11-23T03:00:28.0735835Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0735950Z     traceback.print_stack()
2022-11-23T03:00:28.0736062Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0736173Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0736356Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0736484Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0737040Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0737183Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0737382Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0737510Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0737717Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0737802Z     self.run()
2022-11-23T03:00:28.0737992Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0738207Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0738394Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0738528Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0738730Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0738821Z     self.run()
2022-11-23T03:00:28.0739161Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0739275Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0739465Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0739599Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0739955Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0740070Z     getattr(self, test_name)()
2022-11-23T03:00:28.0740395Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0740515Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0740865Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0740944Z     fn()
2022-11-23T03:00:28.0741291Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0741401Z     getattr(self, test_name)()
2022-11-23T03:00:28.0741753Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0741863Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0742212Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0742297Z     fn()
2022-11-23T03:00:28.0742645Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0742753Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0743105Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0743216Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0743618Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0743717Z     self.run_subtests(
2022-11-23T03:00:28.0744215Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0744333Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0744730Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0744893Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0745133Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0745232Z     self.run_subtests(
2022-11-23T03:00:28.0745585Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0745726Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0746062Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0746211Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0746569Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0746671Z     output = model(*input)
2022-11-23T03:00:28.0747026Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0747217Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0747533Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0747662Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0748026Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0748133Z     output = model(*input)
2022-11-23T03:00:28.0748496Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0748806Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0749113Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0749240Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0749585Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0749693Z     _lazy_init(state, module)
2022-11-23T03:00:28.0750045Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0750202Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0750528Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0750648Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0750991Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0751095Z     _lazy_init(state, module)
2022-11-23T03:00:28.0751415Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0751521Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0752034Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0752163Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0752530Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0752613Z     p_assert(
2022-11-23T03:00:28.0752941Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0753052Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0753376Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0753491Z     traceback.print_stack()
2022-11-23T03:00:28.0753898Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0753995Z     p_assert(
2022-11-23T03:00:28.0754326Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0754434Z     traceback.print_stack()
2022-11-23T03:00:28.0754664Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0754888Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0755106Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0755329Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0755445Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0755798Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0755924Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0756105Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0756288Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0756482Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0756569Z     self.run()
2022-11-23T03:00:28.0756748Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0756878Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0757200Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0757311Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0757647Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0757753Z     getattr(self, test_name)()
2022-11-23T03:00:28.0758090Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0758337Z     fn()
2022-11-23T03:00:28.0758690Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0758805Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0759146Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0759252Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0759368Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0759612Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0759712Z     self.run_subtests(
2022-11-23T03:00:28.0760052Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0760203Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0760403Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0760533Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0760885Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0761185Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0761370Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0761505Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0761860Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0761965Z     output = model(*input)
2022-11-23T03:00:28.0762159Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0762247Z     self.run()
2022-11-23T03:00:28.0762592Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0762720Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0762908Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0763041Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0763581Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0763745Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0764073Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0764188Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0764544Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0764652Z     _lazy_init(state, module)
2022-11-23T03:00:28.0765004Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0765115Z     getattr(self, test_name)()
2022-11-23T03:00:28.0765508Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0765640Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0765986Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0766065Z     fn()
2022-11-23T03:00:28.0766396Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0766509Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0766861Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0766972Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0767341Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0767433Z     p_assert(
2022-11-23T03:00:28.0768125Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0768222Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0768549Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0768663Z     traceback.print_stack()
2022-11-23T03:00:28.0768905Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0769006Z     self.run_subtests(
2022-11-23T03:00:28.0769346Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0769496Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0769850Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0769984Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0770351Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0770457Z     output = model(*input)
2022-11-23T03:00:28.0770929Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0771053Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0771405Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0771563Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0771904Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0772003Z     _lazy_init(state, module)
2022-11-23T03:00:28.0772378Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0772515Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0772832Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0772941Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0773292Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0773380Z     p_assert(
2022-11-23T03:00:28.0773692Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0773795Z     traceback.print_stack()
2022-11-23T03:00:28.0773907Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0774097Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0774227Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0774412Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0774597Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0774792Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0774881Z     self.run()
2022-11-23T03:00:28.0775062Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0775190Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0775301Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0775625Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0775743Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0776081Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0776188Z     getattr(self, test_name)()
2022-11-23T03:00:28.0776378Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0776505Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0777201Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0777291Z     fn()
2022-11-23T03:00:28.0777485Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0777621Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0777983Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0778098Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0778295Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0778388Z     self.run()
2022-11-23T03:00:28.0778738Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0778850Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0779045Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0779179Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0779422Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0779525Z     self.run_subtests(
2022-11-23T03:00:28.0779871Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0779969Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0780313Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0780463Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0780882Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0781002Z     getattr(self, test_name)()
2022-11-23T03:00:28.0781364Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0781506Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0781846Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0781930Z     fn()
2022-11-23T03:00:28.0782293Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0782400Z     output = model(*input)
2022-11-23T03:00:28.0782758Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0782868Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0783188Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0783319Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0783738Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0783851Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0784216Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0784378Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0784622Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0784723Z     self.run_subtests(
2022-11-23T03:00:28.0785230Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0785336Z     _lazy_init(state, module)
2022-11-23T03:00:28.0785664Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0785813Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0786143Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0786271Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0786610Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0786745Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0787065Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0787175Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0787520Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0787625Z     output = model(*input)
2022-11-23T03:00:28.0787985Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0788077Z     p_assert(
2022-11-23T03:00:28.0788381Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0788505Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0788817Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0788929Z     traceback.print_stack()
2022-11-23T03:00:28.0789272Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0789429Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0789768Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0789919Z     _lazy_init(state, module)
2022-11-23T03:00:28.0790253Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0790384Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0790701Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0790810Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0791154Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0791241Z     p_assert(
2022-11-23T03:00:28.0791553Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0791663Z     traceback.print_stack()
2022-11-23T03:00:28.0791883Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0792103Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0792314Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0792577Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0792837Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0793047Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0793254Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0793637Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0793857Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0794073Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0794292Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0794506Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0794715Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0794932Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0795145Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0795358Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0795570Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0795779Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0796007Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0796214Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0796432Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0796639Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0796849Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0797063Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0797279Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0797491Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0797703Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0797964Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0798185Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0798398Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0798611Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0798823Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0799188Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0799572Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0799786Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0799997Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0800215Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0800421Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0800687Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0800902Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0801114Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0801325Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0801539Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0801752Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0801965Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0802336Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0802540Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0802745Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0802949Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0803154Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0803359Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0803562Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0803929Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0804150Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0804360Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0804580Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0804791Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0804999Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0805213Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0805425Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0805637Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0805850Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0806053Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0806313Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0806535Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0806900Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0807107Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0807312Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0807517Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0807721Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0807925Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0808131Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0808336Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0808584Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0808790Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0808995Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0809201Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0809407Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0809613Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0809813Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0810020Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0810224Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0810431Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0810639Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0811109Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0811325Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0811537Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0811747Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0811956Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0812172Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0812391Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0812602Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0812813Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0813023Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0813234Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0813450Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0813544Z dist init r=0, world=4
2022-11-23T03:00:28.0814065Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.0814384Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.0814681Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.0814969Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.0815274Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.0815568Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.0815858Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.0816389Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.0816859Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.0817172Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.0817479Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.0817784Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.0817891Z dist init r=1, world=4
2022-11-23T03:00:28.0818193Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.0818492Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.0818944Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.0819226Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.0819524Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.0819814Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.0820101Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.0820392Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.0820674Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.0820957Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.0821310Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.0821608Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.0821704Z dist init r=2, world=4
2022-11-23T03:00:28.0821989Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.0822272Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.0822555Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.0822840Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.0823199Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.0823488Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.0823773Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.0824060Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.0824349Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.0824639Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.0824931Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.0825219Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.0825315Z dist init r=3, world=4
2022-11-23T03:00:28.0825601Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.0826067Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.0826365Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.0826656Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.0826958Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.0827251Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.0827591Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.0827901Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.0828202Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.0828494Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.0828796Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.0829090Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.0829340Z ok (54.225s)
2022-11-23T03:00:28.0829547Z   test_delayed_reduce_scatter_offload_true_shard_grad_op (__main__.TestParityWithDDP)
2022-11-23T03:00:28.0829889Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 124057
2022-11-23T03:00:28.0830095Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 124058
2022-11-23T03:00:28.0830288Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 124059
2022-11-23T03:00:28.0830490Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 124060
2022-11-23T03:00:28.0830857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:00:28.0831019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:00:28.0831379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:00:28.0831553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:00:28.0832135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:00:28.0832300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:00:28.0832666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:00:28.0832848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:00:28.0833196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:00:28.0833354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:00:28.0833715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:00:28.0833896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:00:28.0834247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:00:28.0834410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:00:28.0834769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:00:28.0835090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:00:28.0835318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:00:28.0835541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:00:28.0835763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:00:28.0836029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:00:28.0836412Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:00:28.0836788Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:00:28.0837341Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:00:28.0837726Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:00:28.0837941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:00:28.0838152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:00:28.0838357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:00:28.0838574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:00:28.0838880Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0839110Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0839334Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0839543Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0840564Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:00:28.0840667Z   warnings.warn(
2022-11-23T03:00:28.0841654Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:00:28.0841753Z   warnings.warn(
2022-11-23T03:00:28.0842733Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:00:28.0842834Z   warnings.warn(
2022-11-23T03:00:28.0843964Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:00:28.0844059Z   warnings.warn(
2022-11-23T03:00:28.0844172Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0844282Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0844646Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0844778Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0845021Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0845164Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0845367Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0845493Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0845695Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0845786Z     self.run()
2022-11-23T03:00:28.0845973Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0846111Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0846321Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0846433Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0846632Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0846721Z     self.run()
2022-11-23T03:00:28.0847063Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0847233Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0847426Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0847560Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0847909Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0848021Z     getattr(self, test_name)()
2022-11-23T03:00:28.0848356Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0848476Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0848839Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0848914Z     fn()
2022-11-23T03:00:28.0849269Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0849383Z     getattr(self, test_name)()
2022-11-23T03:00:28.0849757Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0849848Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0850195Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0850280Z     fn()
2022-11-23T03:00:28.0850625Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0850740Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0851096Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0851210Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0851450Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0851556Z     self.run_subtests(
2022-11-23T03:00:28.0851901Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0852013Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0852357Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0852508Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0852748Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0852849Z     self.run_subtests(
2022-11-23T03:00:28.0853197Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0853337Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0853727Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0853890Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0854257Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0854364Z     output = model(*input)
2022-11-23T03:00:28.0854869Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0855004Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0855302Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0855425Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0855778Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0855883Z     output = model(*input)
2022-11-23T03:00:28.0856245Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0856489Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0857062Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0857191Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0857539Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0857644Z     _lazy_init(state, module)
2022-11-23T03:00:28.0857996Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0858152Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0858650Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0858788Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0859148Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0859255Z     _lazy_init(state, module)
2022-11-23T03:00:28.0859574Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0859687Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0860027Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0860157Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0860528Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0860619Z     p_assert(
2022-11-23T03:00:28.0860951Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0861070Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0861414Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0861504Z     traceback.print_stack()
2022-11-23T03:00:28.0861866Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0861955Z     p_assert(
2022-11-23T03:00:28.0862277Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0862390Z     traceback.print_stack()
2022-11-23T03:00:28.0862505Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0862703Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0862827Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0863091Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0863240Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0863445Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0863537Z     self.run()
2022-11-23T03:00:28.0863743Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0863868Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0864195Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0864317Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0864666Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0864776Z     getattr(self, test_name)()
2022-11-23T03:00:28.0865122Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0865229Z     fn()
2022-11-23T03:00:28.0865568Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0865743Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0866088Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0866199Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0866439Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0866540Z     self.run_subtests(
2022-11-23T03:00:28.0866883Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0867032Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0867388Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0867529Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0867888Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0867997Z     output = model(*input)
2022-11-23T03:00:28.0868312Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0868441Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0868805Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0868969Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0869321Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0869429Z     _lazy_init(state, module)
2022-11-23T03:00:28.0869767Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0869903Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0870374Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0870483Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0870838Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0870925Z     p_assert(
2022-11-23T03:00:28.0871236Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0871347Z     traceback.print_stack()
2022-11-23T03:00:28.0871454Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0871645Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0871771Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0872000Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0872148Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0872342Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0872429Z     self.run()
2022-11-23T03:00:28.0872786Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0872911Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0873245Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0873366Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0873724Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0873825Z     getattr(self, test_name)()
2022-11-23T03:00:28.0874172Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0874259Z     fn()
2022-11-23T03:00:28.0874607Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0874770Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0875114Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0875225Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0875468Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0875569Z     self.run_subtests(
2022-11-23T03:00:28.0875908Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0876057Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0876407Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0876549Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0877067Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0877170Z     output = model(*input)
2022-11-23T03:00:28.0877645Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0877781Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0878144Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0878308Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0878664Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0878767Z     _lazy_init(state, module)
2022-11-23T03:00:28.0879108Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0879241Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0879574Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0879686Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0880051Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0880141Z     p_assert(
2022-11-23T03:00:28.0880460Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0880577Z     traceback.print_stack()
2022-11-23T03:00:28.0880802Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0881027Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0881298Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0881532Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0881650Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0881762Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0881954Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0882085Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0882283Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0882412Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0882602Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0882741Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0882925Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0883066Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0883259Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0883399Z     self.run()
2022-11-23T03:00:28.0883592Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0883682Z     self.run()
2022-11-23T03:00:28.0883872Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0884006Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0884193Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0884319Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0884653Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0884775Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0885108Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0885226Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0885735Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0885843Z     getattr(self, test_name)()
2022-11-23T03:00:28.0886180Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0886281Z     getattr(self, test_name)()
2022-11-23T03:00:28.0886615Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0886696Z     fn()
2022-11-23T03:00:28.0887027Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0887109Z     fn()
2022-11-23T03:00:28.0887456Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0887563Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0887906Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0888005Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0888332Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0888440Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0888774Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0888881Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0889113Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0889213Z     self.run_subtests(
2022-11-23T03:00:28.0889488Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0889585Z     self.run_subtests(
2022-11-23T03:00:28.0889924Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0890069Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0890396Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0890538Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0890877Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0891012Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0891345Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0891471Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0891825Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0891977Z     output = model(*input)
2022-11-23T03:00:28.0892332Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0892434Z     output = model(*input)
2022-11-23T03:00:28.0892794Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0892916Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0893222Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0893339Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0893694Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0894034Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0894405Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0894572Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0894689Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0895045Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0895155Z     _lazy_init(state, module)
2022-11-23T03:00:28.0895503Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0895612Z     _lazy_init(state, module)
2022-11-23T03:00:28.0895820Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0895939Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0896282Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0896414Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0896925Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0897060Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0897243Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0897382Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0897716Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0897829Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0898156Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0898267Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0898695Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0898792Z     self.run()
2022-11-23T03:00:28.0899144Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0899235Z     p_assert(
2022-11-23T03:00:28.0899579Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0899837Z     p_assert(
2022-11-23T03:00:28.0900031Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0900164Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0900489Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0900596Z     traceback.print_stack()
2022-11-23T03:00:28.0900923Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0901036Z     traceback.print_stack()
2022-11-23T03:00:28.0901367Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0901563Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0901917Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0902027Z     getattr(self, test_name)()
2022-11-23T03:00:28.0902373Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0902451Z     fn()
2022-11-23T03:00:28.0902957Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0903063Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0903396Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0903503Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0903739Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0903841Z     self.run_subtests(
2022-11-23T03:00:28.0904344Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0904489Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0904837Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0904977Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0905352Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0905463Z     output = model(*input)
2022-11-23T03:00:28.0905778Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0905909Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0906277Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0906438Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0906792Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0906899Z     _lazy_init(state, module)
2022-11-23T03:00:28.0907386Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0907513Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0907827Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0907935Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0908334Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0908421Z     p_assert(
2022-11-23T03:00:28.0908743Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0908857Z     traceback.print_stack()
2022-11-23T03:00:28.0908969Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0909161Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0909286Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0909470Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0909603Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0909789Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0909876Z     self.run()
2022-11-23T03:00:28.0910060Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0910194Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0910516Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0910682Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0911076Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0911184Z     getattr(self, test_name)()
2022-11-23T03:00:28.0911697Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0911781Z     fn()
2022-11-23T03:00:28.0912136Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0912246Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0912586Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0912697Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0912943Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0913043Z     self.run_subtests(
2022-11-23T03:00:28.0913387Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0913536Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0913886Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0914025Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0914534Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0914639Z     output = model(*input)
2022-11-23T03:00:28.0914942Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0915064Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0915417Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0915578Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0915922Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0916027Z     _lazy_init(state, module)
2022-11-23T03:00:28.0916354Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0916481Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0916797Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0916904Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0917302Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0917395Z     p_assert(
2022-11-23T03:00:28.0917714Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0917824Z     traceback.print_stack()
2022-11-23T03:00:28.0918042Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0918258Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0918471Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0918684Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0918791Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0918982Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0919106Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0919293Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0919475Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0919667Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0919755Z     self.run()
2022-11-23T03:00:28.0919934Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0920062Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0920381Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0920496Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0920834Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0920943Z     getattr(self, test_name)()
2022-11-23T03:00:28.0921282Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0921365Z     fn()
2022-11-23T03:00:28.0921704Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0921814Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0922150Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0922259Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0922492Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0922589Z     self.run_subtests(
2022-11-23T03:00:28.0922916Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0923060Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0923397Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0923535Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0923889Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0923992Z     output = model(*input)
2022-11-23T03:00:28.0924295Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0924418Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0924771Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0924928Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0925266Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0925371Z     _lazy_init(state, module)
2022-11-23T03:00:28.0925743Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0925879Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0926369Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0926483Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0926852Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0926949Z     p_assert(
2022-11-23T03:00:28.0927261Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0927378Z     traceback.print_stack()
2022-11-23T03:00:28.0927496Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0927696Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0927825Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0928018Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0928209Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0928411Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0928497Z     self.run()
2022-11-23T03:00:28.0928612Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0928800Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0928933Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0929265Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0929384Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0929738Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0929857Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0930199Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0930310Z     getattr(self, test_name)()
2022-11-23T03:00:28.0930493Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0930627Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0930965Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0931048Z     fn()
2022-11-23T03:00:28.0931244Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0931325Z     self.run()
2022-11-23T03:00:28.0931847Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0931958Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0932151Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0932287Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0932638Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0932753Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0933072Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0933192Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0933433Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0933533Z     self.run_subtests(
2022-11-23T03:00:28.0933885Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0933994Z     getattr(self, test_name)()
2022-11-23T03:00:28.0934489Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0934680Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0935022Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0935106Z     fn()
2022-11-23T03:00:28.0935444Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0935578Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0935917Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0936024Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0936373Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0936475Z     output = model(*input)
2022-11-23T03:00:28.0937169Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0937290Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0937694Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0937823Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0938063Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0938164Z     self.run_subtests(
2022-11-23T03:00:28.0938534Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0938697Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0939030Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0939179Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0939539Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0939650Z     _lazy_init(state, module)
2022-11-23T03:00:28.0940004Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0940142Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0940484Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0940617Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0940982Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0941083Z     output = model(*input)
2022-11-23T03:00:28.0941411Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0941538Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0941841Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0941973Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0942343Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0942434Z     p_assert(
2022-11-23T03:00:28.0942795Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0942959Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0943287Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0943403Z     traceback.print_stack()
2022-11-23T03:00:28.0943760Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0943932Z     _lazy_init(state, module)
2022-11-23T03:00:28.0944434Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0944565Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0945052Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0945160Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0945543Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0945616Z     p_assert(
2022-11-23T03:00:28.0945940Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0946055Z     traceback.print_stack()
2022-11-23T03:00:28.0946171Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0946368Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0946496Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0946688Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0946878Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0947082Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0947172Z     self.run()
2022-11-23T03:00:28.0947360Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0947493Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0947825Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0947940Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0948289Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0948399Z     getattr(self, test_name)()
2022-11-23T03:00:28.0948752Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0948840Z     fn()
2022-11-23T03:00:28.0949195Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0949456Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0949790Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0950062Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0950307Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0950408Z     self.run_subtests(
2022-11-23T03:00:28.0950750Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0950899Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0951252Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0951396Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0951760Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0951861Z     output = model(*input)
2022-11-23T03:00:28.0952175Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0952304Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0952668Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0952831Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0953183Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0953340Z     _lazy_init(state, module)
2022-11-23T03:00:28.0953687Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0953816Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0954145Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0954256Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0954624Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0954713Z     p_assert(
2022-11-23T03:00:28.0955185Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0955294Z     traceback.print_stack()
2022-11-23T03:00:28.0955512Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0955727Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0955934Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0956196Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0956309Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0956501Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0956628Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0956811Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0956938Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0957132Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0957220Z     self.run()
2022-11-23T03:00:28.0957401Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0957532Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0957856Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0957976Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0958315Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0958416Z     getattr(self, test_name)()
2022-11-23T03:00:28.0958751Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0958994Z     fn()
2022-11-23T03:00:28.0959349Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0959459Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0959802Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0959917Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0960162Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0960260Z     self.run_subtests(
2022-11-23T03:00:28.0960602Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0960752Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0961103Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0961243Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0961608Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0961867Z     output = model(*input)
2022-11-23T03:00:28.0962173Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0962338Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0962702Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0962863Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0963210Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0963314Z     _lazy_init(state, module)
2022-11-23T03:00:28.0963813Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0963945Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0964273Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0964380Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0964748Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0964837Z     p_assert(
2022-11-23T03:00:28.0965229Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0965342Z     traceback.print_stack()
2022-11-23T03:00:28.0965461Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0965666Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0965796Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0965979Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0966118Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0966319Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0966410Z     self.run()
2022-11-23T03:00:28.0966597Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0966736Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0967068Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0967186Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0967538Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0967647Z     getattr(self, test_name)()
2022-11-23T03:00:28.0967995Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0968080Z     fn()
2022-11-23T03:00:28.0968584Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0968689Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0969020Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0969127Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0969359Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0969460Z     self.run_subtests(
2022-11-23T03:00:28.0969978Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0970128Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0970476Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0970619Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0970984Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0971085Z     output = model(*input)
2022-11-23T03:00:28.0971448Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0971584Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0971958Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0972121Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0972632Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0972737Z     _lazy_init(state, module)
2022-11-23T03:00:28.0973064Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0973185Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0973500Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0973608Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0973966Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0974100Z     p_assert(
2022-11-23T03:00:28.0974417Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0974528Z     traceback.print_stack()
2022-11-23T03:00:28.0974641Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0974825Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0974951Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0975134Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0975268Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0975462Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0975550Z     self.run()
2022-11-23T03:00:28.0975733Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0975864Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0976177Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0976297Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0976636Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0976913Z     getattr(self, test_name)()
2022-11-23T03:00:28.0977267Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0977350Z     fn()
2022-11-23T03:00:28.0977690Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0977791Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0978307Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0978422Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0978665Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0978769Z     self.run_subtests(
2022-11-23T03:00:28.0979112Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0979262Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0979612Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0979752Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0980110Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0980217Z     output = model(*input)
2022-11-23T03:00:28.0980598Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0980739Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0981132Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0981273Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0981687Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0981799Z     _lazy_init(state, module)
2022-11-23T03:00:28.0982133Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0982265Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0982592Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0982704Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0983073Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0983229Z     p_assert(
2022-11-23T03:00:28.0983561Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0983673Z     traceback.print_stack()
2022-11-23T03:00:28.0983784Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0983981Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0984112Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0984302Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0984441Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0984643Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0984735Z     self.run()
2022-11-23T03:00:28.0984921Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0985055Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0985388Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0985509Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0986008Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0986114Z     getattr(self, test_name)()
2022-11-23T03:00:28.0986448Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0986531Z     fn()
2022-11-23T03:00:28.0986866Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0986972Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0987310Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0987418Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0987648Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0987749Z     self.run_subtests(
2022-11-23T03:00:28.0988080Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0988224Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0988560Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0988696Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0989046Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0989149Z     output = model(*input)
2022-11-23T03:00:28.0989497Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0989627Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0989986Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0990144Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.0990478Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.0990583Z     _lazy_init(state, module)
2022-11-23T03:00:28.0990909Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.0991035Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.0991349Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.0991456Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0991811Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.0991949Z     p_assert(
2022-11-23T03:00:28.0992261Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.0992370Z     traceback.print_stack()
2022-11-23T03:00:28.0992588Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0992849Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0993061Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0993275Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.0993387Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.0993581Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.0993705Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.0993891Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.0994028Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.0994400Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.0994502Z     self.run()
2022-11-23T03:00:28.0994698Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.0994817Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.0995147Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.0995267Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.0995618Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.0995729Z     getattr(self, test_name)()
2022-11-23T03:00:28.0996081Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.0996170Z     fn()
2022-11-23T03:00:28.0996525Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.0996636Z     test(self, **param_kwargs)
2022-11-23T03:00:28.0996975Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.0997086Z     return func(*args, **kwargs)
2022-11-23T03:00:28.0997327Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.0997427Z     self.run_subtests(
2022-11-23T03:00:28.0997765Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.0997914Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.0998348Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.0998501Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.0998862Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.0998970Z     output = model(*input)
2022-11-23T03:00:28.0999283Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.0999412Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.0999776Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.0999939Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.1000300Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.1000412Z     _lazy_init(state, module)
2022-11-23T03:00:28.1000749Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.1000926Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.1001256Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.1001369Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1001736Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.1001827Z     p_assert(
2022-11-23T03:00:28.1002151Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.1002265Z     traceback.print_stack()
2022-11-23T03:00:28.1002375Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.1002729Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.1002859Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.1003043Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.1003181Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.1003376Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.1003463Z     self.run()
2022-11-23T03:00:28.1003646Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.1003769Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.1004088Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.1004205Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.1004711Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.1004823Z     getattr(self, test_name)()
2022-11-23T03:00:28.1005172Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.1005261Z     fn()
2022-11-23T03:00:28.1005610Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.1005721Z     test(self, **param_kwargs)
2022-11-23T03:00:28.1006070Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.1006182Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1006424Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.1006525Z     self.run_subtests(
2022-11-23T03:00:28.1006866Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.1007015Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.1007558Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.1007705Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.1008054Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.1008157Z     output = model(*input)
2022-11-23T03:00:28.1008461Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.1008586Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.1008939Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.1009099Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.1009440Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.1009542Z     _lazy_init(state, module)
2022-11-23T03:00:28.1009870Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.1010045Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.1010362Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.1010471Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1010822Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.1010963Z     p_assert(
2022-11-23T03:00:28.1011275Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.1011387Z     traceback.print_stack()
2022-11-23T03:00:28.1011499Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.1011864Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.1011996Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.1012189Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.1012326Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.1012528Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.1012612Z     self.run()
2022-11-23T03:00:28.1012804Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.1012937Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.1013268Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.1013389Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.1013739Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.1013849Z     getattr(self, test_name)()
2022-11-23T03:00:28.1014199Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.1014282Z     fn()
2022-11-23T03:00:28.1014788Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.1014895Z     test(self, **param_kwargs)
2022-11-23T03:00:28.1015228Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.1015336Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1015568Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.1015665Z     self.run_subtests(
2022-11-23T03:00:28.1015994Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.1016197Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.1016551Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.1017037Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.1017432Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.1017539Z     output = model(*input)
2022-11-23T03:00:28.1017852Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.1017981Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.1018345Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.1018502Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.1018861Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.1018968Z     _lazy_init(state, module)
2022-11-23T03:00:28.1019385Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.1019516Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.1019997Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.1020109Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1020461Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.1020542Z     p_assert(
2022-11-23T03:00:28.1020854Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.1020963Z     traceback.print_stack()
2022-11-23T03:00:28.1021077Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.1021271Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.1021399Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.1021584Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.1021711Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.1021905Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.1021992Z     self.run()
2022-11-23T03:00:28.1022174Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.1022301Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.1022619Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.1022736Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.1023075Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.1023180Z     getattr(self, test_name)()
2022-11-23T03:00:28.1023514Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.1023598Z     fn()
2022-11-23T03:00:28.1023942Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.1024048Z     test(self, **param_kwargs)
2022-11-23T03:00:28.1024383Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.1024491Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1024724Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.1024815Z     self.run_subtests(
2022-11-23T03:00:28.1025146Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.1025348Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.1025704Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.1025842Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.1026192Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.1026294Z     output = model(*input)
2022-11-23T03:00:28.1026771Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.1026895Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.1027262Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.1027427Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.1027786Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.1027896Z     _lazy_init(state, module)
2022-11-23T03:00:28.1028301Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.1028431Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.1028761Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.1028867Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1029235Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.1029325Z     p_assert(
2022-11-23T03:00:28.1029651Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.1029764Z     traceback.print_stack()
2022-11-23T03:00:28.1029991Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1030221Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1030442Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1030658Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1030776Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.1030975Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.1031110Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.1031232Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.1031417Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.1031557Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.1031759Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.1031850Z     self.run()
2022-11-23T03:00:28.1032048Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.1032179Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.1032371Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.1032503Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.1032685Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.1032976Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.1033299Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.1033416Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.1033606Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.1033693Z     self.run()
2022-11-23T03:00:28.1034080Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.1034194Z     getattr(self, test_name)()
2022-11-23T03:00:28.1034376Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.1034505Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.1034846Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.1034929Z     fn()
2022-11-23T03:00:28.1035242Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.1035357Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.1035699Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.1035801Z     test(self, **param_kwargs)
2022-11-23T03:00:28.1036139Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.1036249Z     getattr(self, test_name)()
2022-11-23T03:00:28.1036584Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.1036745Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1037079Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.1037159Z     fn()
2022-11-23T03:00:28.1037394Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.1037485Z     self.run_subtests(
2022-11-23T03:00:28.1038008Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.1038118Z     test(self, **param_kwargs)
2022-11-23T03:00:28.1038458Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.1038611Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.1038957Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.1039072Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1039425Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.1039560Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.1039802Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.1039903Z     self.run_subtests(
2022-11-23T03:00:28.1040268Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.1040374Z     output = model(*input)
2022-11-23T03:00:28.1040718Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.1040870Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.1041186Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.1041310Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.1041662Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.1041801Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.1042165Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.1042328Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.1042694Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.1042804Z     output = model(*input)
2022-11-23T03:00:28.1043208Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.1043325Z     _lazy_init(state, module)
2022-11-23T03:00:28.1043637Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.1043764Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.1044108Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.1044238Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.1044756Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.1044912Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.1045025Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.1045520Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.1045628Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1046039Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.1046146Z     _lazy_init(state, module)
2022-11-23T03:00:28.1046510Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.1046600Z     p_assert(
2022-11-23T03:00:28.1046940Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.1047069Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.1047261Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.1047390Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.1047716Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.1047831Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1048159Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.1048278Z     traceback.print_stack()
2022-11-23T03:00:28.1048468Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.1048608Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.1048973Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.1049062Z     p_assert(
2022-11-23T03:00:28.1049264Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.1049355Z     self.run()
2022-11-23T03:00:28.1049827Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.1049937Z     traceback.print_stack()
2022-11-23T03:00:28.1050123Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.1050252Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.1050567Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.1050683Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.1051025Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.1051133Z     getattr(self, test_name)()
2022-11-23T03:00:28.1051465Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.1051546Z     fn()
2022-11-23T03:00:28.1051886Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.1052162Z     test(self, **param_kwargs)
2022-11-23T03:00:28.1052547Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.1052667Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1052911Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.1053013Z     self.run_subtests(
2022-11-23T03:00:28.1053356Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.1053507Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.1053860Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.1053999Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.1054357Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.1054466Z     output = model(*input)
2022-11-23T03:00:28.1054783Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.1054960Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.1055328Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.1055489Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.1055845Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.1056105Z     _lazy_init(state, module)
2022-11-23T03:00:28.1056426Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.1056552Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.1057111Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.1057226Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1057586Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.1057677Z     p_assert(
2022-11-23T03:00:28.1057991Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.1058101Z     traceback.print_stack()
2022-11-23T03:00:28.1058211Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.1058400Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.1058525Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.1058710Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.1058844Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.1059039Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.1059128Z     self.run()
2022-11-23T03:00:28.1059480Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.1059618Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.1059951Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.1060071Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.1060422Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.1060531Z     getattr(self, test_name)()
2022-11-23T03:00:28.1060875Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.1060959Z     fn()
2022-11-23T03:00:28.1061306Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.1061415Z     test(self, **param_kwargs)
2022-11-23T03:00:28.1061829Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.1061953Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1062345Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.1062442Z     self.run_subtests(
2022-11-23T03:00:28.1062775Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.1062918Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.1063248Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.1063382Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.1063730Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.1063833Z     output = model(*input)
2022-11-23T03:00:28.1064136Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.1064506Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.1064868Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.1065031Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.1065381Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.1065488Z     _lazy_init(state, module)
2022-11-23T03:00:28.1065827Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.1065959Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.1066288Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.1066405Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1066777Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.1066873Z     p_assert(
2022-11-23T03:00:28.1067193Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.1067307Z     traceback.print_stack()
2022-11-23T03:00:28.1067532Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1067757Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1067973Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1068193Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1068310Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.1068513Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.1068639Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.1068833Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.1068970Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.1069171Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.1069261Z     self.run()
2022-11-23T03:00:28.1069452Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.1069587Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.1069914Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.1070037Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.1070389Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.1070546Z     getattr(self, test_name)()
2022-11-23T03:00:28.1070902Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.1070991Z     fn()
2022-11-23T03:00:28.1071498Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.1071605Z     test(self, **param_kwargs)
2022-11-23T03:00:28.1071932Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.1072040Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1072272Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.1072370Z     self.run_subtests(
2022-11-23T03:00:28.1072699Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.1072847Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.1073187Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.1073371Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.1073718Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.1073822Z     output = model(*input)
2022-11-23T03:00:28.1074124Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.1074249Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.1074600Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.1074757Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.1075104Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.1075210Z     _lazy_init(state, module)
2022-11-23T03:00:28.1075538Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.1075663Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.1075980Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.1076087Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1076443Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.1076529Z     p_assert(
2022-11-23T03:00:28.1076843Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.1076952Z     traceback.print_stack()
2022-11-23T03:00:28.1077059Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.1077255Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.1077381Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.1077568Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.1077700Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.1077894Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.1077981Z     self.run()
2022-11-23T03:00:28.1078161Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.1078460Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.1078791Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.1078910Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.1079258Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.1079414Z     getattr(self, test_name)()
2022-11-23T03:00:28.1079769Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.1079857Z     fn()
2022-11-23T03:00:28.1080206Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.1080320Z     test(self, **param_kwargs)
2022-11-23T03:00:28.1080663Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.1080774Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1081016Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.1081133Z     self.run_subtests(
2022-11-23T03:00:28.1081456Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.1081607Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.1081961Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.1082157Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.1082520Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.1082626Z     output = model(*input)
2022-11-23T03:00:28.1082940Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.1083068Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.1083450Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.1083599Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.1083952Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.1084059Z     _lazy_init(state, module)
2022-11-23T03:00:28.1084399Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.1084527Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.1084852Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.1084965Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1085332Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.1085422Z     p_assert(
2022-11-23T03:00:28.1085742Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.1085849Z     traceback.print_stack()
2022-11-23T03:00:28.1085966Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.1086319Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.1086450Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.1086631Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.1086764Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.1086958Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.1087040Z     self.run()
2022-11-23T03:00:28.1087224Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.1087353Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.1087672Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.1087788Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.1088173Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.1088286Z     getattr(self, test_name)()
2022-11-23T03:00:28.1088623Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.1088703Z     fn()
2022-11-23T03:00:28.1089047Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.1089155Z     test(self, **param_kwargs)
2022-11-23T03:00:28.1089486Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.1089595Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1089828Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.1089926Z     self.run_subtests(
2022-11-23T03:00:28.1090257Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.1090401Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.1090809Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.1090945Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.1091295Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.1091398Z     output = model(*input)
2022-11-23T03:00:28.1091703Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.1091827Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.1092179Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.1092332Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.1092678Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.1092785Z     _lazy_init(state, module)
2022-11-23T03:00:28.1093170Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.1093297Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.1093610Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.1093717Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1094071Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.1094152Z     p_assert(
2022-11-23T03:00:28.1094463Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.1094744Z     traceback.print_stack()
2022-11-23T03:00:28.1094875Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.1095088Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.1095205Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.1095395Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.1095527Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.1095730Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.1095821Z     self.run()
2022-11-23T03:00:28.1096007Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.1096140Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.1096473Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.1096594Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.1097199Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.1097314Z     getattr(self, test_name)()
2022-11-23T03:00:28.1097674Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.1097759Z     fn()
2022-11-23T03:00:28.1098112Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.1098222Z     test(self, **param_kwargs)
2022-11-23T03:00:28.1098567Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.1098679Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1098917Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.1099012Z     self.run_subtests(
2022-11-23T03:00:28.1099504Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.1099649Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.1100221Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.1100362Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.1100726Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.1100832Z     output = model(*input)
2022-11-23T03:00:28.1101146Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.1101268Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.1101632Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.1101795Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.1102157Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.1102268Z     _lazy_init(state, module)
2022-11-23T03:00:28.1102605Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.1102738Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.1103218Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.1103321Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1103676Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.1103762Z     p_assert(
2022-11-23T03:00:28.1104076Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.1104184Z     traceback.print_stack()
2022-11-23T03:00:28.1104406Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1104621Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1104839Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1105217Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1105336Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.1105532Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.1105662Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.1105849Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.1105987Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.1106191Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.1106283Z     self.run()
2022-11-23T03:00:28.1106514Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.1106660Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.1106993Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.1107114Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.1107467Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.1107577Z     getattr(self, test_name)()
2022-11-23T03:00:28.1108073Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.1108150Z     fn()
2022-11-23T03:00:28.1108488Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.1108594Z     test(self, **param_kwargs)
2022-11-23T03:00:28.1108926Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.1109081Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1109317Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.1109415Z     self.run_subtests(
2022-11-23T03:00:28.1109746Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.1109885Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.1110222Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.1110358Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.1110708Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.1110812Z     output = model(*input)
2022-11-23T03:00:28.1111173Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.1111302Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.1111653Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.1111810Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.1112319Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.1112428Z     _lazy_init(state, module)
2022-11-23T03:00:28.1112765Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.1112895Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.1113221Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.1113337Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1113701Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.1113794Z     p_assert(
2022-11-23T03:00:28.1114116Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.1114230Z     traceback.print_stack()
2022-11-23T03:00:28.1114348Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.1114544Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.1114674Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.1114790Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.1115129Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.1115259Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.1115502Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.1115599Z     self.run()
2022-11-23T03:00:28.1115793Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.1115917Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.1116102Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.1116401Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.1116585Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.1116723Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.1117057Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.1117175Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.1117375Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.1117465Z     self.run()
2022-11-23T03:00:28.1117819Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.1117982Z     getattr(self, test_name)()
2022-11-23T03:00:28.1118169Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.1118302Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.1118653Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.1118738Z     fn()
2022-11-23T03:00:28.1119060Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.1119335Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.1119678Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.1119782Z     test(self, **param_kwargs)
2022-11-23T03:00:28.1120111Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.1120217Z     getattr(self, test_name)()
2022-11-23T03:00:28.1120555Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.1120662Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1120997Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.1121078Z     fn()
2022-11-23T03:00:28.1121310Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.1121408Z     self.run_subtests(
2022-11-23T03:00:28.1121745Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.1121852Z     test(self, **param_kwargs)
2022-11-23T03:00:28.1122183Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.1122332Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.1122666Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.1122773Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1123110Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.1123245Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.1123469Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.1123565Z     self.run_subtests(
2022-11-23T03:00:28.1123914Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.1124017Z     output = model(*input)
2022-11-23T03:00:28.1124388Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.1124542Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.1124850Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.1124975Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.1125308Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.1125443Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.1125792Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.1125951Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.1126301Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.1126407Z     output = model(*input)
2022-11-23T03:00:28.1126751Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.1127072Z     _lazy_init(state, module)
2022-11-23T03:00:28.1127404Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.1127522Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.1127862Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.1127992Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.1128357Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.1128520Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.1128850Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.1128961Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1129317Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.1129425Z     _lazy_init(state, module)
2022-11-23T03:00:28.1129793Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.1129884Z     p_assert(
2022-11-23T03:00:28.1130375Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.1130502Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.1130815Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.1130925Z     traceback.print_stack()
2022-11-23T03:00:28.1131237Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.1131345Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1131700Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.1131848Z     p_assert(
2022-11-23T03:00:28.1132167Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.1132446Z     traceback.print_stack()
2022-11-23T03:00:28.1132579Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.1132763Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.1132887Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.1133079Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.1133219Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.1133468Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.1133566Z     self.run()
2022-11-23T03:00:28.1133757Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.1133893Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.1134213Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.1134334Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.1134693Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.1134792Z     getattr(self, test_name)()
2022-11-23T03:00:28.1135156Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.1135234Z     fn()
2022-11-23T03:00:28.1135590Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.1135701Z     test(self, **param_kwargs)
2022-11-23T03:00:28.1136033Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.1136231Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1136471Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.1136572Z     self.run_subtests(
2022-11-23T03:00:28.1137109Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.1137272Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.1137621Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.1137748Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.1138111Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.1138222Z     output = model(*input)
2022-11-23T03:00:28.1138531Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.1138662Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.1139029Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.1139192Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.1139547Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.1139654Z     _lazy_init(state, module)
2022-11-23T03:00:28.1139987Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.1140120Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.1140453Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.1140564Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1140937Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.1141026Z     p_assert(
2022-11-23T03:00:28.1141348Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.1141461Z     traceback.print_stack()
2022-11-23T03:00:28.1141683Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1141904Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1142126Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1142339Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1142529Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.1142739Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.1142874Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.1143063Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.1143196Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.1143397Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.1143488Z     self.run()
2022-11-23T03:00:28.1143675Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.1143809Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.1144145Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.1144268Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.1144623Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.1144728Z     getattr(self, test_name)()
2022-11-23T03:00:28.1145059Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.1145398Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.1145480Z     fn()
2022-11-23T03:00:28.1145994Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.1146103Z     test(self, **param_kwargs)
2022-11-23T03:00:28.1146300Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.1146424Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.1146770Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.1146884Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1147079Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.1147217Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.1147461Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.1147562Z     self.run_subtests(
2022-11-23T03:00:28.1147764Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.1147849Z     self.run()
2022-11-23T03:00:28.1148192Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.1148343Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.1148535Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.1148669Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.1149028Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.1149167Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.1149498Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.1149613Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.1150130Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.1150234Z     output = model(*input)
2022-11-23T03:00:28.1150574Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.1150680Z     getattr(self, test_name)()
2022-11-23T03:00:28.1151153Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.1151297Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.1151681Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.1151769Z     fn()
2022-11-23T03:00:28.1152136Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.1152303Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.1152658Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.1152768Z     test(self, **param_kwargs)
2022-11-23T03:00:28.1153119Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.1153227Z     _lazy_init(state, module)
2022-11-23T03:00:28.1153573Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.1153679Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1154023Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.1154153Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.1154461Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.1154561Z     self.run_subtests(
2022-11-23T03:00:28.1154890Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.1155154Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1155484Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.1155623Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.1155974Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.1156061Z     p_assert(
2022-11-23T03:00:28.1156411Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.1156550Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.1156862Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.1156972Z     traceback.print_stack()
2022-11-23T03:00:28.1157323Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.1157420Z     output = model(*input)
2022-11-23T03:00:28.1157722Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.1157847Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.1158198Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.1158355Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.1158706Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.1158813Z     _lazy_init(state, module)
2022-11-23T03:00:28.1159139Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.1159260Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.1159576Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.1159855Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1160220Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.1160309Z     p_assert(
2022-11-23T03:00:28.1160633Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.1160746Z     traceback.print_stack()
2022-11-23T03:00:28.1160908Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.1161110Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.1161244Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.1161433Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.1161571Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.1161773Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.1161867Z     self.run()
2022-11-23T03:00:28.1162057Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.1162183Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.1162670Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.1162786Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.1163128Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.1163284Z     getattr(self, test_name)()
2022-11-23T03:00:28.1163621Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.1163702Z     fn()
2022-11-23T03:00:28.1164040Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.1164141Z     test(self, **param_kwargs)
2022-11-23T03:00:28.1164474Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.1164754Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1164996Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.1165097Z     self.run_subtests(
2022-11-23T03:00:28.1165442Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.1165593Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.1165947Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.1166080Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.1166442Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.1166551Z     output = model(*input)
2022-11-23T03:00:28.1166865Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.1166992Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.1167357Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.1167524Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.1167883Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.1167989Z     _lazy_init(state, module)
2022-11-23T03:00:28.1168323Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.1168455Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.1168780Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.1168891Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1169256Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.1169346Z     p_assert(
2022-11-23T03:00:28.1169672Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.1169780Z     traceback.print_stack()
2022-11-23T03:00:28.1169945Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.1170151Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.1170442Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.1170624Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.1170759Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.1170952Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.1171039Z     self.run()
2022-11-23T03:00:28.1171216Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.1171345Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.1171665Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.1171780Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.1172122Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.1172277Z     getattr(self, test_name)()
2022-11-23T03:00:28.1172616Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.1172692Z     fn()
2022-11-23T03:00:28.1173030Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.1173135Z     test(self, **param_kwargs)
2022-11-23T03:00:28.1173464Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.1173570Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1173980Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.1174081Z     self.run_subtests(
2022-11-23T03:00:28.1174425Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.1174573Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.1174923Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.1175061Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.1175423Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.1175531Z     output = model(*input)
2022-11-23T03:00:28.1175845Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.1175973Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.1176339Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.1176500Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.1177190Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.1177305Z     _lazy_init(state, module)
2022-11-23T03:00:28.1177640Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.1177766Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.1178080Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.1178187Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1178542Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.1178798Z     p_assert(
2022-11-23T03:00:28.1179121Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.1179305Z     traceback.print_stack()
2022-11-23T03:00:28.1179540Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1179767Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1179989Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1180211Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1180328Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.1180522Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.1180666Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.1180841Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.1180980Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.1181184Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.1181276Z     self.run()
2022-11-23T03:00:28.1181480Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.1181660Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.1181988Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.1182109Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.1182464Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.1182575Z     getattr(self, test_name)()
2022-11-23T03:00:28.1182925Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.1183009Z     fn()
2022-11-23T03:00:28.1183361Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.1183475Z     test(self, **param_kwargs)
2022-11-23T03:00:28.1183814Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.1183930Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1184169Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.1184270Z     self.run_subtests(
2022-11-23T03:00:28.1184614Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.1184764Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.1185111Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.1185251Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.1185611Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.1185719Z     output = model(*input)
2022-11-23T03:00:28.1186033Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.1186167Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.1186681Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.1186838Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.1187183Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.1187287Z     _lazy_init(state, module)
2022-11-23T03:00:28.1187608Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.1187734Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.1188094Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.1188211Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1188574Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.1188662Z     p_assert(
2022-11-23T03:00:28.1188974Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.1189084Z     traceback.print_stack()
2022-11-23T03:00:28.1189191Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.1189381Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.1189505Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.1189687Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.1189819Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.1190015Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.1190104Z     self.run()
2022-11-23T03:00:28.1190275Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.1190461Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.1190589Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.1190909Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.1191027Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.1191218Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.1191344Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.1191683Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.1191783Z     getattr(self, test_name)()
2022-11-23T03:00:28.1191967Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.1192101Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.1192440Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.1192522Z     fn()
2022-11-23T03:00:28.1192716Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.1192805Z     self.run()
2022-11-23T03:00:28.1193198Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.1193309Z     test(self, **param_kwargs)
2022-11-23T03:00:28.1193495Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.1193624Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.1193961Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.1194072Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1194386Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.1194506Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.1194733Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.1194831Z     self.run_subtests(
2022-11-23T03:00:28.1195348Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.1195460Z     getattr(self, test_name)()
2022-11-23T03:00:28.1195799Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.1195948Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.1196292Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.1196377Z     fn()
2022-11-23T03:00:28.1196775Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.1196928Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.1197277Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.1197389Z     test(self, **param_kwargs)
2022-11-23T03:00:28.1197753Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.1197859Z     output = model(*input)
2022-11-23T03:00:28.1198202Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.1198313Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1198623Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.1198756Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.1199005Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.1199148Z     self.run_subtests(
2022-11-23T03:00:28.1199514Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.1199675Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.1200013Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.1200162Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.1200510Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.1200618Z     _lazy_init(state, module)
2022-11-23T03:00:28.1200974Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.1201114Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.1201458Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.1201597Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.1201955Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.1202062Z     output = model(*input)
2022-11-23T03:00:28.1202381Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.1202493Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1202809Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.1202936Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.1203308Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.1203401Z     p_assert(
2022-11-23T03:00:28.1203770Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.1203929Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.1204250Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.1204366Z     traceback.print_stack()
2022-11-23T03:00:28.1204721Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.1204830Z     _lazy_init(state, module)
2022-11-23T03:00:28.1205171Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.1205301Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.1205671Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.1205790Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1206162Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.1206246Z     p_assert(
2022-11-23T03:00:28.1206569Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.1206682Z     traceback.print_stack()
2022-11-23T03:00:28.1206799Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.1207000Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.1207130Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.1207319Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.1207458Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.1207668Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.1207799Z     self.run()
2022-11-23T03:00:28.1207987Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.1208138Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.1208454Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.1208590Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.1208922Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.1209032Z     getattr(self, test_name)()
2022-11-23T03:00:28.1209379Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.1209463Z     fn()
2022-11-23T03:00:28.1209819Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.1209929Z     test(self, **param_kwargs)
2022-11-23T03:00:28.1210278Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.1210389Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1210623Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.1210723Z     self.run_subtests(
2022-11-23T03:00:28.1211125Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.1211276Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.1211630Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.1211769Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.1212136Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.1212246Z     output = model(*input)
2022-11-23T03:00:28.1212554Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.1212695Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.1213049Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.1213213Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.1213564Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.1213672Z     _lazy_init(state, module)
2022-11-23T03:00:28.1214012Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.1214143Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.1214518Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.1214635Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1215005Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.1215094Z     p_assert(
2022-11-23T03:00:28.1215416Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.1215529Z     traceback.print_stack()
2022-11-23T03:00:28.1215754Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1215978Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1216194Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1216422Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1216541Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.1216995Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.1217129Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.1217319Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.1217459Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.1217660Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.1217745Z     self.run()
2022-11-23T03:00:28.1217936Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.1218068Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.1218409Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.1218531Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.1218889Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.1219003Z     getattr(self, test_name)()
2022-11-23T03:00:28.1219354Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.1219435Z     fn()
2022-11-23T03:00:28.1219791Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.1219902Z     test(self, **param_kwargs)
2022-11-23T03:00:28.1220248Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.1220360Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1220604Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.1220704Z     self.run_subtests(
2022-11-23T03:00:28.1221048Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.1221196Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.1221553Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.1221694Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.1222061Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.1222167Z     output = model(*input)
2022-11-23T03:00:28.1222481Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.1222611Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.1222977Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.1223206Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.1223577Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.1223688Z     _lazy_init(state, module)
2022-11-23T03:00:28.1224028Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.1224156Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.1224479Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.1224592Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1224960Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.1225044Z     p_assert(
2022-11-23T03:00:28.1225370Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.1225491Z     traceback.print_stack()
2022-11-23T03:00:28.1225608Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.1225872Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.1226003Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.1226194Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.1226327Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.1226528Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.1226620Z     self.run()
2022-11-23T03:00:28.1226809Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.1226942Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.1227274Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.1227395Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.1227767Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.1227861Z     getattr(self, test_name)()
2022-11-23T03:00:28.1228210Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.1228295Z     fn()
2022-11-23T03:00:28.1228657Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.1228769Z     test(self, **param_kwargs)
2022-11-23T03:00:28.1229116Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.1229228Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1229467Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.1229562Z     self.run_subtests(
2022-11-23T03:00:28.1229905Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.1230059Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.1230412Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.1230552Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.1230916Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.1231024Z     output = model(*input)
2022-11-23T03:00:28.1231339Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.1231462Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.1231831Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.1232041Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.1232408Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.1232521Z     _lazy_init(state, module)
2022-11-23T03:00:28.1232860Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.1232992Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.1233319Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.1233425Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1233790Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.1233880Z     p_assert(
2022-11-23T03:00:28.1234207Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.1234324Z     traceback.print_stack()
2022-11-23T03:00:28.1234440Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.1234693Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.1234823Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.1235009Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.1235147Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.1235349Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.1235439Z     self.run()
2022-11-23T03:00:28.1235628Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.1235763Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.1236095Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.1236213Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.1236563Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.1236678Z     getattr(self, test_name)()
2022-11-23T03:00:28.1237026Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.1237112Z     fn()
2022-11-23T03:00:28.1237464Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.1237574Z     test(self, **param_kwargs)
2022-11-23T03:00:28.1237920Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.1238026Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1238269Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.1238370Z     self.run_subtests(
2022-11-23T03:00:28.1238715Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.1238868Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.1239222Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.1239363Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.1239729Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.1239830Z     output = model(*input)
2022-11-23T03:00:28.1240145Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.1240274Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.1240644Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.1240851Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.1241220Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.1241329Z     _lazy_init(state, module)
2022-11-23T03:00:28.1241667Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.1241791Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.1242116Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.1242230Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1242598Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.1242687Z     p_assert(
2022-11-23T03:00:28.1243015Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.1243130Z     traceback.print_stack()
2022-11-23T03:00:28.1243247Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.1243489Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.1243620Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.1243810Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.1243949Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.1244150Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.1244241Z     self.run()
2022-11-23T03:00:28.1244430Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.1244565Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.1244893Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.1245018Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.1245370Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.1245484Z     getattr(self, test_name)()
2022-11-23T03:00:28.1245830Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.1245915Z     fn()
2022-11-23T03:00:28.1246270Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.1246373Z     test(self, **param_kwargs)
2022-11-23T03:00:28.1246720Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.1246835Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1247076Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.1247180Z     self.run_subtests(
2022-11-23T03:00:28.1247522Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.1247674Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.1248028Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.1248168Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.1248525Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.1248634Z     output = model(*input)
2022-11-23T03:00:28.1248948Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.1249075Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.1249488Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.1249660Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.1250022Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.1250129Z     _lazy_init(state, module)
2022-11-23T03:00:28.1250460Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.1250594Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.1250920Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.1251033Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1251398Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.1251488Z     p_assert(
2022-11-23T03:00:28.1251819Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.1251931Z     traceback.print_stack()
2022-11-23T03:00:28.1252202Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1252426Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1252641Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1252862Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1252980Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.1253180Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.1253309Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.1253494Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.1253634Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.1253839Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.1253932Z     self.run()
2022-11-23T03:00:28.1254120Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.1254254Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.1254587Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.1254707Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.1255054Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.1255165Z     getattr(self, test_name)()
2022-11-23T03:00:28.1255512Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.1255597Z     fn()
2022-11-23T03:00:28.1255956Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.1256067Z     test(self, **param_kwargs)
2022-11-23T03:00:28.1256419Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.1256532Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1256947Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.1257055Z     self.run_subtests(
2022-11-23T03:00:28.1257402Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.1257553Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.1257904Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.1258043Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.1258474Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.1258598Z     output = model(*input)
2022-11-23T03:00:28.1258911Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.1259043Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.1259411Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.1259573Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.1259929Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.1260036Z     _lazy_init(state, module)
2022-11-23T03:00:28.1260376Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.1260506Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.1260834Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.1261010Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1261381Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.1261470Z     p_assert(
2022-11-23T03:00:28.1261798Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.1261913Z     traceback.print_stack()
2022-11-23T03:00:28.1262032Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.1262229Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.1262354Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.1262550Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.1262687Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.1262891Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.1262986Z     self.run()
2022-11-23T03:00:28.1263179Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.1263311Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.1263639Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.1263762Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.1264113Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.1264224Z     getattr(self, test_name)()
2022-11-23T03:00:28.1264570Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.1264654Z     fn()
2022-11-23T03:00:28.1265013Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.1265123Z     test(self, **param_kwargs)
2022-11-23T03:00:28.1265236Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.1265584Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.1265697Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1265940Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.1266041Z     self.run_subtests(
2022-11-23T03:00:28.1266238Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.1266367Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.1266712Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.1266859Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.1267097Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.1267246Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.1267603Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.1267743Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.1267946Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.1268038Z     self.run()
2022-11-23T03:00:28.1268402Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.1268503Z     output = model(*input)
2022-11-23T03:00:28.1268694Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.1268827Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.1269149Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.1269278Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.1269657Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.1269777Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.1270135Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.1270301Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.1270650Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.1270759Z     getattr(self, test_name)()
2022-11-23T03:00:28.1271116Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.1271223Z     _lazy_init(state, module)
2022-11-23T03:00:28.1271573Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.1271661Z     fn()
2022-11-23T03:00:28.1272004Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.1272130Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.1272482Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.1272592Z     test(self, **param_kwargs)
2022-11-23T03:00:28.1272919Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.1273031Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1273375Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.1273486Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1273848Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.1273940Z     p_assert(
2022-11-23T03:00:28.1274182Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.1274283Z     self.run_subtests(
2022-11-23T03:00:28.1274608Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.1274721Z     traceback.print_stack()
2022-11-23T03:00:28.1275058Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.1275207Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.1275550Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.1275691Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.1276138Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.1276260Z     output = model(*input)
2022-11-23T03:00:28.1276576Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.1276705Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.1277067Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.1277228Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.1277586Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.1277688Z     _lazy_init(state, module)
2022-11-23T03:00:28.1278025Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.1278158Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.1278487Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.1278659Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1279027Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.1279116Z     p_assert(
2022-11-23T03:00:28.1279436Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.1279551Z     traceback.print_stack()
2022-11-23T03:00:28.1279668Z   File "<string>", line 1, in <module>
2022-11-23T03:00:28.1279864Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-11-23T03:00:28.1279994Z     exitcode = _main(fd, parent_sentinel)
2022-11-23T03:00:28.1280185Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-11-23T03:00:28.1280327Z     return self._bootstrap(parent_sentinel)
2022-11-23T03:00:28.1280529Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-11-23T03:00:28.1280617Z     self.run()
2022-11-23T03:00:28.1280826Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-11-23T03:00:28.1280939Z     self._target(*self._args, **self._kwargs)
2022-11-23T03:00:28.1281269Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run
2022-11-23T03:00:28.1281400Z     self.run_test(test_name, pipe)
2022-11-23T03:00:28.1281818Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:00:28.1281908Z     getattr(self, test_name)()
2022-11-23T03:00:28.1282257Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:00:28.1282337Z     fn()
2022-11-23T03:00:28.1282694Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test
2022-11-23T03:00:28.1282808Z     test(self, **param_kwargs)
2022-11-23T03:00:28.1283154Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:00:28.1283268Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1283511Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter
2022-11-23T03:00:28.1283613Z     self.run_subtests(
2022-11-23T03:00:28.1283956Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests
2022-11-23T03:00:28.1284100Z     test_fn(*test_args, **test_kwargs, **subtest_kwargs)
2022-11-23T03:00:28.1284450Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity
2022-11-23T03:00:28.1284590Z     fsdp_loss = self._train_for_several_steps(
2022-11-23T03:00:28.1285005Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps
2022-11-23T03:00:28.1285121Z     output = model(*input)
2022-11-23T03:00:28.1285439Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl
2022-11-23T03:00:28.1285567Z     return forward_call(*input, **kwargs)
2022-11-23T03:00:28.1285934Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward
2022-11-23T03:00:28.1286091Z     args, kwargs = _root_pre_forward(self, self, *args, **kwargs)
2022-11-23T03:00:28.1286445Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward
2022-11-23T03:00:28.1286552Z     _lazy_init(state, module)
2022-11-23T03:00:28.1286890Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init
2022-11-23T03:00:28.1287025Z     handle.init_flat_param_attributes()
2022-11-23T03:00:28.1287355Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-11-23T03:00:28.1287517Z     return func(*args, **kwargs)
2022-11-23T03:00:28.1287887Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes
2022-11-23T03:00:28.1287972Z     p_assert(
2022-11-23T03:00:28.1288298Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2022-11-23T03:00:28.1288411Z     traceback.print_stack()
2022-11-23T03:00:28.1288638Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1288863Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1289080Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1289306Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1289533Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1289747Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1289963Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1290174Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1290388Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1290607Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1290820Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1291032Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1291247Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1291459Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1291678Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1291889Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1292101Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1292314Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1292527Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1292739Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1293044Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1293270Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1293480Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1293692Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1293906Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1294117Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1294330Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1294542Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1294754Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1294971Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1295179Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1295440Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1295651Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1295863Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1296077Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1296290Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1296501Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1296890Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1297115Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1297333Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1297546Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1297758Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1297970Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1298183Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1298394Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1298608Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1298821Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1299029Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1299244Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1299456Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1299672Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1299884Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1300092Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1300304Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1300515Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1300794Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1301015Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1301229Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1301442Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1301656Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1301868Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1302081Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1302292Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1302500Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1302714Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1302927Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1303197Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1303407Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1303620Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1303832Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1304046Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1304256Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1304460Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1304678Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1304893Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1305107Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1305317Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1305525Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1305739Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1305951Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1306160Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1306370Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1306584Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1306801Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1307012Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1307226Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1307437Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1307645Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1307853Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1308065Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1308330Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1308550Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1308763Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1309116Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1309333Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1309595Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T03:00:28.1331256Z dist init r=0, world=4
2022-11-23T03:00:28.1331654Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.1331989Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.1332306Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.1332710Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.1333049Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.1333373Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.1333686Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.1333999Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.1334309Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.1334615Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.1334923Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.1335227Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor.
2022-11-23T03:00:28.1335342Z dist init r=1, world=4
2022-11-23T03:00:28.1335692Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.1335990Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.1336298Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.1336605Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.1337201Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.1337617Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.1337939Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.1338235Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.1338539Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.1338867Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.1339164Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.1339474Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor.
2022-11-23T03:00:28.1339643Z dist init r=2, world=4
2022-11-23T03:00:28.1339930Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.1340235Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.1340545Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.1340866Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.1341154Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.1341461Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.1341767Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.1342071Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.1342374Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.1342680Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.1342987Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.1343291Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:2 after the FSDP constructor.
2022-11-23T03:00:28.1343386Z dist init r=3, world=4
2022-11-23T03:00:28.1343716Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.1344034Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.1344391Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.1344707Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.1345013Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.1345325Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.1345632Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.1345940Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.1346244Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.1346597Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.1346881Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.1347193Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:3 after the FSDP constructor.
2022-11-23T03:00:28.1347303Z ok (41.397s)
2022-11-23T03:00:28.1347920Z   test_mixture_of_experts_offload_false_no_shard_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1348479Z   test_mixture_of_experts_offload_false_none_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1349047Z   test_mixture_of_experts_offload_false_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1349602Z   test_mixture_of_experts_offload_true_no_shard_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1350150Z   test_mixture_of_experts_offload_true_none_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1350715Z   test_mixture_of_experts_offload_true_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1351289Z   test_mixture_of_experts_with_delay_before_free_offload_false_no_shard (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1351853Z   test_mixture_of_experts_with_delay_before_free_offload_false_none (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1352437Z   test_mixture_of_experts_with_delay_before_free_offload_false_shard_grad_op (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1353060Z   test_mixture_of_experts_with_delay_before_free_offload_true_no_shard (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1353632Z   test_mixture_of_experts_with_delay_before_free_offload_true_none (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1354214Z   test_mixture_of_experts_with_delay_before_free_offload_true_shard_grad_op (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1354762Z   test_nested_always_wrap_model_offload_false_no_shard_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1355329Z   test_nested_always_wrap_model_offload_false_none_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1355956Z   test_nested_always_wrap_model_offload_false_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1356519Z   test_nested_always_wrap_model_offload_true_no_shard_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1357071Z   test_nested_always_wrap_model_offload_true_none_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1357643Z   test_nested_always_wrap_model_offload_true_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1358179Z   test_nested_wrapped_model_offload_false_no_shard (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1358702Z   test_nested_wrapped_model_offload_false_none (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1359249Z   test_nested_wrapped_model_offload_false_shard_grad_op (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1359782Z   test_nested_wrapped_model_offload_true_no_shard (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1360304Z   test_nested_wrapped_model_offload_true_none (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1360850Z   test_nested_wrapped_model_offload_true_shard_grad_op (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1361453Z   test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_no_shard (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1362041Z   test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_none (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1362701Z   test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1363307Z   test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1363895Z   test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1364503Z   test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_shard_grad_op (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1365027Z   test_transformer_offload_false_no_shard_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1365617Z   test_transformer_offload_false_none_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1366166Z   test_transformer_offload_false_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1366705Z   test_transformer_offload_true_no_shard_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1367245Z   test_transformer_offload_true_none_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1367797Z   test_transformer_offload_true_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:28.1367820Z 
2022-11-23T03:00:28.1368089Z ----------------------------------------------------------------------
2022-11-23T03:00:28.1368209Z Ran 59 tests in 214.056s
2022-11-23T03:00:28.1368229Z 
2022-11-23T03:00:28.1368339Z OK (skipped=54)
2022-11-23T03:00:28.1368357Z 
2022-11-23T03:00:28.1368481Z Generating XML reports...
2022-11-23T03:00:28.1368927Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParityWithDDP-20221123025653.xml
2022-11-23T03:00:28.1369320Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestHooks-20221123025653.xml
2022-11-23T03:00:28.1369725Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestNoGrad-20221123025653.xml
2022-11-23T03:00:28.1370136Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParamInit-20221123025653.xml
2022-11-23T03:00:28.1370156Z 
2022-11-23T03:00:28.1370586Z ##[endgroup]
2022-11-23T03:00:28.1371049Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_core (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_core_pcw1rry1)
2022-11-23T03:00:28.1371069Z 
2022-11-23T03:00:28.3013340Z 
2022-11-23T03:00:28.3013533Z real	3m42.041s
2022-11-23T03:00:28.3013886Z user	9m43.616s
2022-11-23T03:00:28.3014136Z sys	5m34.694s
2022-11-23T03:00:28.3014420Z + for f in test/distributed/fsdp/*.py
2022-11-23T03:00:28.3014913Z + python test/run_test.py --verbose -i distributed/fsdp/test_fsdp_exec_order.py
2022-11-23T03:00:30.6927698Z Ignoring disabled issues:  []
2022-11-23T03:00:30.7458167Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:00:30.7458852Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:00:30.7459220Z Selected tests:
2022-11-23T03:00:30.7459507Z  distributed/fsdp/test_fsdp_exec_order.py
2022-11-23T03:00:30.7484865Z Prioritized test from test file changes.
2022-11-23T03:00:30.7485224Z reordering tests for PR:
2022-11-23T03:00:30.7485515Z prioritized: []
2022-11-23T03:00:30.7486016Z the rest: ['distributed/fsdp/test_fsdp_exec_order.py']
2022-11-23T03:00:30.7486249Z 
2022-11-23T03:00:30.7486804Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:00:30.7487776Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:00:30.7493657Z parallel (file granularity) tests:
2022-11-23T03:00:30.7494168Z  
2022-11-23T03:00:30.7494436Z serial (file granularity) tests:
2022-11-23T03:00:30.7494774Z  distributed/fsdp/test_fsdp_exec_order.py
2022-11-23T03:00:33.0640449Z Ignoring disabled issues:  []
2022-11-23T03:00:33.0649690Z Ignoring disabled issues:  []
2022-11-23T03:00:33.5335314Z Running distributed/fsdp/test_fsdp_exec_order.py ... [2022-11-23 03:00:33.532859]
2022-11-23T03:00:33.5336612Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_exec_order.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:00:33.533363]
2022-11-23T03:00:37.8000507Z 
2022-11-23T03:00:37.8001282Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_exec_order
2022-11-23T03:00:37.8002385Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_exec_order (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_exec_order_87zrrsvv)
2022-11-23T03:00:37.8002785Z 
2022-11-23T03:00:37.8002870Z Running tests...
2022-11-23T03:00:37.8003476Z ----------------------------------------------------------------------
2022-11-23T03:00:37.8004012Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order
2022-11-23T03:00:37.8004686Z   test_invalid_first_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder)
2022-11-23T03:00:37.8005326Z Tests that FSDP errors if the all-gather order differs across ranks ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:00:37.8005991Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.771s)
2022-11-23T03:00:37.8006518Z   test_invalid_first_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder)
2022-11-23T03:00:37.8007317Z Tests that FSDP errors if the all-gather order differs across ranks ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:37.8007974Z   test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD_iters_before_path_change_1 (__main__.TestFSDPExecOrder)
2022-11-23T03:00:37.8008681Z Tests that FSDP warns the user if the all-gather order changes after ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:00:37.8009308Z   test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD_iters_before_path_change_3 (__main__.TestFSDPExecOrder)
2022-11-23T03:00:37.8010108Z Tests that FSDP warns the user if the all-gather order changes after ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:00:37.8010768Z   test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_iters_before_path_change_1 (__main__.TestFSDPExecOrder)
2022-11-23T03:00:37.8011897Z Tests that FSDP warns the user if the all-gather order changes after ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:00:37.8012564Z   test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_iters_before_path_change_3 (__main__.TestFSDPExecOrder)
2022-11-23T03:00:37.8013387Z Tests that FSDP warns the user if the all-gather order changes after ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:00:37.8014311Z   test_train_eval_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:00:37.8015281Z   test_train_eval_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:00:37.8015716Z 
2022-11-23T03:00:37.8015982Z ----------------------------------------------------------------------
2022-11-23T03:00:37.8016396Z Ran 8 tests in 1.783s
2022-11-23T03:00:37.8016563Z 
2022-11-23T03:00:37.8016780Z OK (skipped=8)
2022-11-23T03:00:37.8017298Z 
2022-11-23T03:00:37.8017486Z Generating XML reports...
2022-11-23T03:00:37.8018126Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order/TEST-TestFSDPExecOrder-20221123030035.xml
2022-11-23T03:00:37.8018482Z 
2022-11-23T03:00:37.8018744Z ##[endgroup]
2022-11-23T03:00:37.8019414Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_exec_order (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_exec_order_87zrrsvv)
2022-11-23T03:00:37.8019775Z 
2022-11-23T03:00:38.1768918Z 
2022-11-23T03:00:38.1769434Z real	0m9.875s
2022-11-23T03:00:38.1769841Z user	0m17.423s
2022-11-23T03:00:38.1769994Z sys	0m13.662s
2022-11-23T03:00:38.1770294Z + for f in test/distributed/fsdp/*.py
2022-11-23T03:00:38.1770922Z + python test/run_test.py --verbose -i distributed/fsdp/test_fsdp_flatten_params.py
2022-11-23T03:00:40.5603717Z Ignoring disabled issues:  []
2022-11-23T03:00:40.6133323Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:00:40.6133804Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:00:40.6134155Z Selected tests:
2022-11-23T03:00:40.6134461Z  distributed/fsdp/test_fsdp_flatten_params.py
2022-11-23T03:00:40.6160088Z Prioritized test from test file changes.
2022-11-23T03:00:40.6160605Z reordering tests for PR:
2022-11-23T03:00:40.6161181Z prioritized: []
2022-11-23T03:00:40.6161647Z the rest: ['distributed/fsdp/test_fsdp_flatten_params.py']
2022-11-23T03:00:40.6161846Z 
2022-11-23T03:00:40.6162494Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:00:40.6163341Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:00:40.6168559Z parallel (file granularity) tests:
2022-11-23T03:00:40.6168940Z  
2022-11-23T03:00:40.6169265Z serial (file granularity) tests:
2022-11-23T03:00:40.6169720Z  distributed/fsdp/test_fsdp_flatten_params.py
2022-11-23T03:00:42.9655713Z Ignoring disabled issues:  []
2022-11-23T03:00:42.9686491Z Ignoring disabled issues:  []
2022-11-23T03:00:43.4108791Z Running distributed/fsdp/test_fsdp_flatten_params.py ... [2022-11-23 03:00:43.410214]
2022-11-23T03:00:43.4109696Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_flatten_params.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:00:43.410681]
2022-11-23T03:00:47.5815086Z 
2022-11-23T03:00:47.5816013Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_flatten_params
2022-11-23T03:00:47.5817522Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_flatten_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_flatten_params_nd7m1qhk)
2022-11-23T03:00:47.5817948Z 
2022-11-23T03:00:47.5818072Z Running tests...
2022-11-23T03:00:47.5818609Z ----------------------------------------------------------------------
2022-11-23T03:00:47.5819182Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_flatten_params
2022-11-23T03:00:47.5819643Z   test_empty_module (__main__.TestFlattenParams)
2022-11-23T03:00:47.5820099Z Tests flattening an empty module (i.e. one without any parameters). ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:00:47.5820780Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.725s)
2022-11-23T03:00:47.5821244Z   test_flat_param_shard_metadata (__main__.TestFlattenParams)
2022-11-23T03:00:47.5822076Z Tests that ``FlatParameter`` shard metadata are computed as expected. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.004s)
2022-11-23T03:00:47.5822757Z   test_flatten_nothing (__main__.TestFlattenParams)
2022-11-23T03:00:47.5823450Z Tests that constructing a ``FlatParamHandle`` with no parameters ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:47.5824008Z   test_numel_with_shared_params (__main__.TestFlattenParams)
2022-11-23T03:00:47.5824738Z Tests that numel is preserved after flattening when there are shared ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:47.5825293Z   test_numel_without_shared_params (__main__.TestFlattenParams)
2022-11-23T03:00:47.5826004Z Tests that numel is preserved after flattening when there are no shared ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:47.5826580Z   test_output_with_shared_params (__main__.TestFlattenParams)
2022-11-23T03:00:47.5827320Z Tests a forward pass after flattening when there are shared parameters ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:47.5827910Z   test_output_without_shared_params (__main__.TestFlattenParams)
2022-11-23T03:00:47.5828625Z Tests a forward pass after flattening when there are no shared ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:47.5829135Z   test_partial_flattening (__main__.TestFlattenParams)
2022-11-23T03:00:47.5829812Z Tests flattening some submodules but not others. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:47.5830361Z   test_pnorm_after_step_with_shared_params (__main__.TestFlattenParams)
2022-11-23T03:00:47.5831108Z Tests for parameter Frobenius norm parity after an optimizer step when ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:47.5831474Z 
2022-11-23T03:00:47.5831748Z ----------------------------------------------------------------------
2022-11-23T03:00:47.5832086Z Ran 9 tests in 1.735s
2022-11-23T03:00:47.5832255Z 
2022-11-23T03:00:47.5832368Z OK (skipped=9)
2022-11-23T03:00:47.5832527Z 
2022-11-23T03:00:47.5832632Z Generating XML reports...
2022-11-23T03:00:47.5833254Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_flatten_params/TEST-TestFlattenParams-20221123030045.xml
2022-11-23T03:00:47.5833622Z 
2022-11-23T03:00:47.5833949Z ##[endgroup]
2022-11-23T03:00:47.5834648Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_flatten_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_flatten_params_nd7m1qhk)
2022-11-23T03:00:47.5835055Z 
2022-11-23T03:00:48.0006622Z 
2022-11-23T03:00:48.0007030Z real	0m9.824s
2022-11-23T03:00:48.0007345Z user	0m18.329s
2022-11-23T03:00:48.0007585Z sys	0m15.108s
2022-11-23T03:00:48.0007893Z + for f in test/distributed/fsdp/*.py
2022-11-23T03:00:48.0008534Z + python test/run_test.py --verbose -i distributed/fsdp/test_fsdp_freezing_weights.py
2022-11-23T03:00:50.4197512Z Ignoring disabled issues:  []
2022-11-23T03:00:50.4728048Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:00:50.4728658Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:00:50.4729003Z Selected tests:
2022-11-23T03:00:50.4729185Z  distributed/fsdp/test_fsdp_freezing_weights.py
2022-11-23T03:00:50.4756785Z Prioritized test from test file changes.
2022-11-23T03:00:50.4757146Z reordering tests for PR:
2022-11-23T03:00:50.4757406Z prioritized: []
2022-11-23T03:00:50.4757972Z the rest: ['distributed/fsdp/test_fsdp_freezing_weights.py']
2022-11-23T03:00:50.4758457Z 
2022-11-23T03:00:50.4759010Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:00:50.4759960Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:00:50.4766680Z parallel (file granularity) tests:
2022-11-23T03:00:50.4767010Z  
2022-11-23T03:00:50.4767264Z serial (file granularity) tests:
2022-11-23T03:00:50.4767588Z  distributed/fsdp/test_fsdp_freezing_weights.py
2022-11-23T03:00:52.7754126Z Ignoring disabled issues:  []
2022-11-23T03:00:52.8405422Z Ignoring disabled issues:  []
2022-11-23T03:00:53.2072680Z Running distributed/fsdp/test_fsdp_freezing_weights.py ... [2022-11-23 03:00:53.206576]
2022-11-23T03:00:53.2074958Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_freezing_weights.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:00:53.207072]
2022-11-23T03:00:57.4710482Z 
2022-11-23T03:00:57.4711854Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_freezing_weights
2022-11-23T03:00:57.4712950Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_freezing_weights (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_freezing_weights__57d_e4k)
2022-11-23T03:00:57.4713347Z 
2022-11-23T03:00:57.4713456Z Running tests...
2022-11-23T03:00:57.4714037Z ----------------------------------------------------------------------
2022-11-23T03:00:57.4714498Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights
2022-11-23T03:00:57.4715299Z   test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:00:57.4715979Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.763s)
2022-11-23T03:00:57.4716984Z   test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:57.4718112Z   test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:57.4719510Z   test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:57.4720698Z   test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:57.4721840Z   test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:57.4722987Z   test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:57.4724135Z   test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:00:57.4724731Z 
2022-11-23T03:00:57.4725004Z ----------------------------------------------------------------------
2022-11-23T03:00:57.4725320Z Ran 8 tests in 1.771s
2022-11-23T03:00:57.4725485Z 
2022-11-23T03:00:57.4725607Z OK (skipped=8)
2022-11-23T03:00:57.4725764Z 
2022-11-23T03:00:57.4725892Z Generating XML reports...
2022-11-23T03:00:57.4726614Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights/TEST-TestFreezingWeights-20221123030055.xml
2022-11-23T03:00:57.4726880Z 
2022-11-23T03:00:57.4727190Z ##[endgroup]
2022-11-23T03:00:57.4727844Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_freezing_weights (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_freezing_weights__57d_e4k)
2022-11-23T03:00:57.4728231Z 
2022-11-23T03:00:57.8269665Z 
2022-11-23T03:00:57.8269857Z real	0m9.826s
2022-11-23T03:00:57.8270115Z user	0m18.067s
2022-11-23T03:00:57.8270551Z sys	0m15.583s
2022-11-23T03:00:57.8270977Z + for f in test/distributed/fsdp/*.py
2022-11-23T03:00:57.8271584Z + python test/run_test.py --verbose -i distributed/fsdp/test_fsdp_fx.py
2022-11-23T03:01:00.2234886Z Ignoring disabled issues:  []
2022-11-23T03:01:00.2765651Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:01:00.2766235Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:01:00.2766610Z Selected tests:
2022-11-23T03:01:00.2766909Z  distributed/fsdp/test_fsdp_fx.py
2022-11-23T03:01:00.2793357Z Prioritized test from test file changes.
2022-11-23T03:01:00.2794740Z reordering tests for PR:
2022-11-23T03:01:00.2795499Z prioritized: []
2022-11-23T03:01:00.2796116Z the rest: ['distributed/fsdp/test_fsdp_fx.py']
2022-11-23T03:01:00.2796437Z 
2022-11-23T03:01:00.2796879Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:01:00.2797920Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:01:00.2800138Z parallel (file granularity) tests:
2022-11-23T03:01:00.2800619Z  
2022-11-23T03:01:00.2801055Z serial (file granularity) tests:
2022-11-23T03:01:00.2801702Z  distributed/fsdp/test_fsdp_fx.py
2022-11-23T03:01:02.6073624Z Ignoring disabled issues:  []
2022-11-23T03:01:02.6078300Z Ignoring disabled issues:  []
2022-11-23T03:01:03.0188747Z Running distributed/fsdp/test_fsdp_fx.py ... [2022-11-23 03:01:03.018224]
2022-11-23T03:01:03.0189880Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_fx.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:01:03.018652]
2022-11-23T03:01:07.1131233Z 
2022-11-23T03:01:07.1131913Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_fx
2022-11-23T03:01:07.1133318Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_fx (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_fx_mrn4ze0l)
2022-11-23T03:01:07.1133682Z 
2022-11-23T03:01:07.1133800Z Running tests...
2022-11-23T03:01:07.1134368Z ----------------------------------------------------------------------
2022-11-23T03:01:07.1134902Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_fx
2022-11-23T03:01:07.1135358Z   test_symbolic_tracing_outputs (__main__.TestSymbolicTracing)
2022-11-23T03:01:07.1135888Z test ``execution_info.module_forward_order`` and ``execution_info.module_to_execution_infos`` ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:01:07.1136583Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.721s)
2022-11-23T03:01:07.1137555Z 
2022-11-23T03:01:07.1137875Z ----------------------------------------------------------------------
2022-11-23T03:01:07.1138192Z Ran 1 test in 1.721s
2022-11-23T03:01:07.1138385Z 
2022-11-23T03:01:07.1138463Z OK (skipped=1)
2022-11-23T03:01:07.1138637Z 
2022-11-23T03:01:07.1138733Z Generating XML reports...
2022-11-23T03:01:07.1139305Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_fx/TEST-TestSymbolicTracing-20221123030104.xml
2022-11-23T03:01:07.1139716Z 
2022-11-23T03:01:07.1140022Z ##[endgroup]
2022-11-23T03:01:07.1140625Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_fx (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_fx_mrn4ze0l)
2022-11-23T03:01:07.1140947Z 
2022-11-23T03:01:07.4716566Z 
2022-11-23T03:01:07.4717077Z real	0m9.645s
2022-11-23T03:01:07.4717484Z user	0m17.423s
2022-11-23T03:01:07.4717743Z sys	0m15.065s
2022-11-23T03:01:07.4718033Z + for f in test/distributed/fsdp/*.py
2022-11-23T03:01:07.4718656Z + python test/run_test.py --verbose -i distributed/fsdp/test_fsdp_grad_acc.py
2022-11-23T03:01:09.8473664Z Ignoring disabled issues:  []
2022-11-23T03:01:09.9002599Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:01:09.9003160Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:01:09.9003538Z Selected tests:
2022-11-23T03:01:09.9003834Z  distributed/fsdp/test_fsdp_grad_acc.py
2022-11-23T03:01:09.9031755Z Prioritized test from test file changes.
2022-11-23T03:01:09.9032391Z reordering tests for PR:
2022-11-23T03:01:09.9032736Z prioritized: []
2022-11-23T03:01:09.9033261Z the rest: ['distributed/fsdp/test_fsdp_grad_acc.py']
2022-11-23T03:01:09.9033468Z 
2022-11-23T03:01:09.9034023Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:01:09.9034983Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:01:09.9041159Z parallel (file granularity) tests:
2022-11-23T03:01:09.9041479Z  
2022-11-23T03:01:09.9041829Z serial (file granularity) tests:
2022-11-23T03:01:09.9042151Z  distributed/fsdp/test_fsdp_grad_acc.py
2022-11-23T03:01:12.1574316Z Ignoring disabled issues:  []
2022-11-23T03:01:12.1836920Z Ignoring disabled issues:  []
2022-11-23T03:01:12.5959319Z Running distributed/fsdp/test_fsdp_grad_acc.py ... [2022-11-23 03:01:12.595225]
2022-11-23T03:01:12.5960500Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_grad_acc.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:01:12.595667]
2022-11-23T03:01:16.6843369Z 
2022-11-23T03:01:16.6843986Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_grad_acc
2022-11-23T03:01:16.6845168Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_grad_acc (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_grad_acc_u36dpx6y)
2022-11-23T03:01:16.6845567Z 
2022-11-23T03:01:16.6845660Z Running tests...
2022-11-23T03:01:16.6846229Z ----------------------------------------------------------------------
2022-11-23T03:01:16.6846795Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc
2022-11-23T03:01:16.6847473Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestGradAcc)
2022-11-23T03:01:16.6848088Z Tests gradient accumulation. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:01:16.6848705Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.704s)
2022-11-23T03:01:16.6849696Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestGradAcc)
2022-11-23T03:01:16.6850539Z Tests gradient accumulation. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:01:16.6851257Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestGradAcc)
2022-11-23T03:01:16.6852070Z Tests gradient accumulation. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:01:16.6852779Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestGradAcc)
2022-11-23T03:01:16.6853617Z Tests gradient accumulation. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:01:16.6854328Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestGradAcc)
2022-11-23T03:01:16.6855149Z Tests gradient accumulation. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:01:16.6855852Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestGradAcc)
2022-11-23T03:01:16.6856969Z Tests gradient accumulation. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:01:16.6857826Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestGradAcc)
2022-11-23T03:01:16.6858673Z Tests gradient accumulation. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:01:16.6859375Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestGradAcc)
2022-11-23T03:01:16.6860287Z Tests gradient accumulation. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:01:16.6860948Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestGradAcc)
2022-11-23T03:01:16.6861793Z Tests gradient accumulation. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:01:16.6862511Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestGradAcc)
2022-11-23T03:01:16.6863459Z Tests gradient accumulation. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:01:16.6864035Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestGradAcc)
2022-11-23T03:01:16.6864940Z Tests gradient accumulation. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:01:16.6865655Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestGradAcc)
2022-11-23T03:01:16.6866488Z Tests gradient accumulation. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:01:16.6866825Z 
2022-11-23T03:01:16.6867101Z ----------------------------------------------------------------------
2022-11-23T03:01:16.6867424Z Ran 12 tests in 1.721s
2022-11-23T03:01:16.6867592Z 
2022-11-23T03:01:16.6867708Z OK (skipped=12)
2022-11-23T03:01:16.6867879Z 
2022-11-23T03:01:16.6868000Z Generating XML reports...
2022-11-23T03:01:16.6868676Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc/TEST-TestGradAcc-20221123030114.xml
2022-11-23T03:01:16.6868924Z 
2022-11-23T03:01:16.6869246Z ##[endgroup]
2022-11-23T03:01:16.6869860Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_grad_acc (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_grad_acc_u36dpx6y)
2022-11-23T03:01:16.6870226Z 
2022-11-23T03:01:17.0421438Z 
2022-11-23T03:01:17.0422058Z real	0m9.570s
2022-11-23T03:01:17.0422397Z user	0m17.548s
2022-11-23T03:01:17.0422670Z sys	0m15.837s
2022-11-23T03:01:17.0422954Z + for f in test/distributed/fsdp/*.py
2022-11-23T03:01:17.0423634Z + python test/run_test.py --verbose -i distributed/fsdp/test_fsdp_ignored_modules.py
2022-11-23T03:01:19.4680692Z Ignoring disabled issues:  []
2022-11-23T03:01:19.5211207Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:01:19.5211808Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:01:19.5212161Z Selected tests:
2022-11-23T03:01:19.5212451Z  distributed/fsdp/test_fsdp_ignored_modules.py
2022-11-23T03:01:19.5241839Z Prioritized test from test file changes.
2022-11-23T03:01:19.5242473Z reordering tests for PR:
2022-11-23T03:01:19.5242804Z prioritized: []
2022-11-23T03:01:19.5243328Z the rest: ['distributed/fsdp/test_fsdp_ignored_modules.py']
2022-11-23T03:01:19.5243562Z 
2022-11-23T03:01:19.5244109Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:01:19.5245383Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:01:19.5250735Z parallel (file granularity) tests:
2022-11-23T03:01:19.5251263Z  
2022-11-23T03:01:19.5251550Z serial (file granularity) tests:
2022-11-23T03:01:19.5251900Z  distributed/fsdp/test_fsdp_ignored_modules.py
2022-11-23T03:01:21.8734574Z Ignoring disabled issues:  []
2022-11-23T03:01:21.9209494Z Ignoring disabled issues:  []
2022-11-23T03:01:22.3929143Z Running distributed/fsdp/test_fsdp_ignored_modules.py ... [2022-11-23 03:01:22.392329]
2022-11-23T03:01:22.3930255Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_ignored_modules.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:01:22.392764]
2022-11-23T03:01:26.5658118Z 
2022-11-23T03:01:26.5659027Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_ignored_modules
2022-11-23T03:01:26.5660007Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_ignored_modules (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_ignored_modules_ett2kyrc)
2022-11-23T03:01:26.5660707Z 
2022-11-23T03:01:26.5660821Z Running tests...
2022-11-23T03:01:26.5661384Z ----------------------------------------------------------------------
2022-11-23T03:01:26.5661944Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules
2022-11-23T03:01:26.5662490Z   test_diff_ignored_modules_across_ranks_pass_ignored_modules_to_root_False (__main__.TestFSDPIgnoredModules)
2022-11-23T03:01:26.5663011Z Tests ignoring different modules across ranks. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:01:26.5663641Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.768s)
2022-11-23T03:01:26.5664202Z   test_diff_ignored_modules_across_ranks_pass_ignored_modules_to_root_True (__main__.TestFSDPIgnoredModules)
2022-11-23T03:01:26.5664963Z Tests ignoring different modules across ranks. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:01:26.5665511Z   test_ignored_modules_invalid (__main__.TestFSDPIgnoredModules)
2022-11-23T03:01:26.5666216Z Tests that passing an FSDP module as an ignored module or the ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:01:26.5666811Z   test_ignored_modules_nested (__main__.TestFSDPIgnoredModules)
2022-11-23T03:01:26.5667611Z Tests that passing a module with nested FSDP modules does not ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:01:26.5668174Z   test_ignored_modules_transformer (__main__.TestFSDPIgnoredModules)
2022-11-23T03:01:26.5668890Z Tests that ignored modules' parameters are not flattened for a ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:01:26.5669277Z 
2022-11-23T03:01:26.5669552Z ----------------------------------------------------------------------
2022-11-23T03:01:26.5669899Z Ran 5 tests in 1.774s
2022-11-23T03:01:26.5670069Z 
2022-11-23T03:01:26.5670186Z OK (skipped=5)
2022-11-23T03:01:26.5670326Z 
2022-11-23T03:01:26.5670457Z Generating XML reports...
2022-11-23T03:01:26.5671110Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules/TEST-TestFSDPIgnoredModules-20221123030124.xml
2022-11-23T03:01:26.5671497Z 
2022-11-23T03:01:26.5671844Z ##[endgroup]
2022-11-23T03:01:26.5672467Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_ignored_modules (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_ignored_modules_ett2kyrc)
2022-11-23T03:01:26.5672852Z 
2022-11-23T03:01:26.9209396Z 
2022-11-23T03:01:26.9209804Z real	0m9.879s
2022-11-23T03:01:26.9210118Z user	0m17.830s
2022-11-23T03:01:26.9210368Z sys	0m16.513s
2022-11-23T03:01:26.9211060Z + for f in test/distributed/fsdp/*.py
2022-11-23T03:01:26.9211749Z + python test/run_test.py --verbose -i distributed/fsdp/test_fsdp_input.py
2022-11-23T03:01:29.3032402Z Ignoring disabled issues:  []
2022-11-23T03:01:29.3563597Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:01:29.3564200Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:01:29.3564537Z Selected tests:
2022-11-23T03:01:29.3564836Z  distributed/fsdp/test_fsdp_input.py
2022-11-23T03:01:29.3591974Z Prioritized test from test file changes.
2022-11-23T03:01:29.3592320Z reordering tests for PR:
2022-11-23T03:01:29.3592608Z prioritized: []
2022-11-23T03:01:29.3593118Z the rest: ['distributed/fsdp/test_fsdp_input.py']
2022-11-23T03:01:29.3593357Z 
2022-11-23T03:01:29.3593922Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:01:29.3594857Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:01:29.3600516Z parallel (file granularity) tests:
2022-11-23T03:01:29.3600811Z  
2022-11-23T03:01:29.3601058Z serial (file granularity) tests:
2022-11-23T03:01:29.3601380Z  distributed/fsdp/test_fsdp_input.py
2022-11-23T03:01:31.6654137Z Ignoring disabled issues:  []
2022-11-23T03:01:31.6796034Z Ignoring disabled issues:  []
2022-11-23T03:01:32.0348865Z Running distributed/fsdp/test_fsdp_input.py ... [2022-11-23 03:01:32.034206]
2022-11-23T03:01:32.0349709Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_input.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:01:32.034659]
2022-11-23T03:01:36.1940907Z 
2022-11-23T03:01:36.1941619Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_input
2022-11-23T03:01:36.1942648Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_input (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_input_2fw0yafn)
2022-11-23T03:01:36.1943032Z 
2022-11-23T03:01:36.1943217Z Running tests...
2022-11-23T03:01:36.1943702Z ----------------------------------------------------------------------
2022-11-23T03:01:36.1944262Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_input
2022-11-23T03:01:36.1944665Z   test_input_type_dict (__main__.TestInput)
2022-11-23T03:01:36.1945088Z Test FSDP with input being a list or a dict, only single GPU. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:01:36.1945740Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.751s)
2022-11-23T03:01:36.1946153Z   test_input_type_list (__main__.TestInput)
2022-11-23T03:01:36.1946834Z Test FSDP with input being a list or a dict, only single GPU. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:01:36.1947211Z 
2022-11-23T03:01:36.1947480Z ----------------------------------------------------------------------
2022-11-23T03:01:36.1947810Z Ran 2 tests in 1.753s
2022-11-23T03:01:36.1947978Z 
2022-11-23T03:01:36.1948070Z OK (skipped=2)
2022-11-23T03:01:36.1948226Z 
2022-11-23T03:01:36.1948359Z Generating XML reports...
2022-11-23T03:01:36.1948934Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_input/TEST-TestInput-20221123030134.xml
2022-11-23T03:01:36.1949270Z 
2022-11-23T03:01:36.1949600Z ##[endgroup]
2022-11-23T03:01:36.1950144Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_input (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_input_2fw0yafn)
2022-11-23T03:01:36.1950496Z 
2022-11-23T03:01:36.5543747Z 
2022-11-23T03:01:36.5544282Z real	0m9.633s
2022-11-23T03:01:36.5544600Z user	0m17.676s
2022-11-23T03:01:36.5545240Z sys	0m14.900s
2022-11-23T03:01:36.5545565Z + for f in test/distributed/fsdp/*.py
2022-11-23T03:01:36.5546212Z + python test/run_test.py --verbose -i distributed/fsdp/test_fsdp_memory.py
2022-11-23T03:01:38.9597575Z Ignoring disabled issues:  []
2022-11-23T03:01:39.0134607Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:01:39.0135647Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:01:39.0136007Z Selected tests:
2022-11-23T03:01:39.0136275Z  distributed/fsdp/test_fsdp_memory.py
2022-11-23T03:01:39.0161172Z Prioritized test from test file changes.
2022-11-23T03:01:39.0161879Z reordering tests for PR:
2022-11-23T03:01:39.0162431Z prioritized: []
2022-11-23T03:01:39.0163043Z the rest: ['distributed/fsdp/test_fsdp_memory.py']
2022-11-23T03:01:39.0163260Z 
2022-11-23T03:01:39.0163830Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:01:39.0165171Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:01:39.0168385Z parallel (file granularity) tests:
2022-11-23T03:01:39.0168923Z  
2022-11-23T03:01:39.0169473Z serial (file granularity) tests:
2022-11-23T03:01:39.0170032Z  distributed/fsdp/test_fsdp_memory.py
2022-11-23T03:01:41.3031055Z Ignoring disabled issues:  []
2022-11-23T03:01:41.3161276Z Ignoring disabled issues:  []
2022-11-23T03:01:41.7158336Z Running distributed/fsdp/test_fsdp_memory.py ... [2022-11-23 03:01:41.715188]
2022-11-23T03:01:41.7159365Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_memory.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:01:41.715621]
2022-11-23T03:01:45.9178177Z 
2022-11-23T03:01:45.9178938Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_memory
2022-11-23T03:01:45.9179958Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_memory (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_memory_5kzeaqzr)
2022-11-23T03:01:45.9180362Z 
2022-11-23T03:01:45.9180480Z Running tests...
2022-11-23T03:01:45.9181036Z ----------------------------------------------------------------------
2022-11-23T03:01:45.9181607Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_memory
2022-11-23T03:01:45.9182112Z   test_fsdp_memory_ckpt_ckpt (__main__.TestFSDPMemory) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:01:45.9182780Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.759s)
2022-11-23T03:01:45.9183597Z   test_fsdp_memory_ckpt_no_ckpt (__main__.TestFSDPMemory) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:01:45.9183979Z 
2022-11-23T03:01:45.9184252Z ----------------------------------------------------------------------
2022-11-23T03:01:45.9184579Z Ran 2 tests in 1.762s
2022-11-23T03:01:45.9184756Z 
2022-11-23T03:01:45.9184870Z OK (skipped=2)
2022-11-23T03:01:45.9185037Z 
2022-11-23T03:01:45.9185163Z Generating XML reports...
2022-11-23T03:01:45.9185759Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_memory/TEST-TestFSDPMemory-20221123030143.xml
2022-11-23T03:01:45.9186089Z 
2022-11-23T03:01:45.9186405Z ##[endgroup]
2022-11-23T03:01:45.9187012Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_memory (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_memory_5kzeaqzr)
2022-11-23T03:01:45.9187373Z 
2022-11-23T03:01:46.2727784Z 
2022-11-23T03:01:46.2728258Z real	0m9.718s
2022-11-23T03:01:46.2728660Z user	0m17.760s
2022-11-23T03:01:46.2728917Z sys	0m15.270s
2022-11-23T03:01:46.2729850Z + for f in test/distributed/fsdp/*.py
2022-11-23T03:01:46.2730391Z + python test/run_test.py --verbose -i distributed/fsdp/test_fsdp_meta.py
2022-11-23T03:01:48.6554919Z Ignoring disabled issues:  []
2022-11-23T03:01:48.7085752Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:01:48.7086360Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:01:48.7086734Z Selected tests:
2022-11-23T03:01:48.7087033Z  distributed/fsdp/test_fsdp_meta.py
2022-11-23T03:01:48.7112317Z Prioritized test from test file changes.
2022-11-23T03:01:48.7112660Z reordering tests for PR:
2022-11-23T03:01:48.7112976Z prioritized: []
2022-11-23T03:01:48.7113458Z the rest: ['distributed/fsdp/test_fsdp_meta.py']
2022-11-23T03:01:48.7113674Z 
2022-11-23T03:01:48.7114237Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:01:48.7115194Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:01:48.7119094Z parallel (file granularity) tests:
2022-11-23T03:01:48.7119355Z  
2022-11-23T03:01:48.7119871Z serial (file granularity) tests:
2022-11-23T03:01:48.7120252Z  distributed/fsdp/test_fsdp_meta.py
2022-11-23T03:01:51.0467285Z Ignoring disabled issues:  []
2022-11-23T03:01:51.0558204Z Ignoring disabled issues:  []
2022-11-23T03:01:51.5294545Z Running distributed/fsdp/test_fsdp_meta.py ... [2022-11-23 03:01:51.528799]
2022-11-23T03:01:51.5296012Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_meta.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:01:51.529330]
2022-11-23T03:01:55.7870522Z 
2022-11-23T03:01:55.7871136Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_meta
2022-11-23T03:01:55.7872450Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_meta (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_meta_m478qbwc)
2022-11-23T03:01:55.7872856Z 
2022-11-23T03:01:55.7872924Z Running tests...
2022-11-23T03:01:55.7873547Z ----------------------------------------------------------------------
2022-11-23T03:01:55.7874128Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_meta
2022-11-23T03:01:55.7874649Z   test_bad_arg_meta (__main__.TestFSDPWithMetaDevice) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:01:55.7875338Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.764s)
2022-11-23T03:01:55.7875943Z   test_bad_arg_torchdistx (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2022-11-23T03:01:55.7876856Z   test_nested_model_with_meta_device_default_init_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:01:55.7877843Z   test_nested_model_with_meta_device_default_init_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:01:55.7878835Z   test_nested_model_with_meta_device_reset_params_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:01:55.7879801Z   test_nested_model_with_meta_device_reset_params_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:01:55.7880893Z   test_nested_model_with_torchdistX_default_init_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2022-11-23T03:01:55.7881558Z   test_nested_model_with_torchdistX_default_init_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2022-11-23T03:01:55.7882236Z   test_nested_model_with_torchdistX_init_fn_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2022-11-23T03:01:55.7882946Z   test_nested_model_with_torchdistX_init_fn_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2022-11-23T03:01:55.7883857Z   test_simple_model_with_meta_device_default_init (__main__.TestFSDPWithMetaDevice) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:01:55.7884799Z   test_simple_model_with_meta_device_reset_params (__main__.TestFSDPWithMetaDevice) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:01:55.7885692Z   test_simple_model_with_torchdistX_default_init (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2022-11-23T03:01:55.7886374Z   test_simple_model_with_torchdistX_init_fn (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2022-11-23T03:01:55.7886732Z 
2022-11-23T03:01:55.7887012Z ----------------------------------------------------------------------
2022-11-23T03:01:55.7887347Z Ran 14 tests in 1.772s
2022-11-23T03:01:55.7887513Z 
2022-11-23T03:01:55.7887605Z OK (skipped=14)
2022-11-23T03:01:55.7887768Z 
2022-11-23T03:01:55.7887895Z Generating XML reports...
2022-11-23T03:01:55.7888532Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_meta/TEST-TestFSDPWithMetaDevice-20221123030153.xml
2022-11-23T03:01:55.7888906Z 
2022-11-23T03:01:55.7889208Z ##[endgroup]
2022-11-23T03:01:55.7889808Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_meta (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_meta_m478qbwc)
2022-11-23T03:01:55.7890165Z 
2022-11-23T03:01:56.1464941Z 
2022-11-23T03:01:56.1465480Z real	0m9.874s
2022-11-23T03:01:56.1466006Z user	0m18.173s
2022-11-23T03:01:56.1466222Z sys	0m15.660s
2022-11-23T03:01:56.1466585Z + for f in test/distributed/fsdp/*.py
2022-11-23T03:01:56.1467078Z + python test/run_test.py --verbose -i distributed/fsdp/test_fsdp_misc.py
2022-11-23T03:01:58.5886852Z Ignoring disabled issues:  []
2022-11-23T03:01:58.6428074Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:01:58.6428692Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:01:58.6429039Z Selected tests:
2022-11-23T03:01:58.6429352Z  distributed/fsdp/test_fsdp_misc.py
2022-11-23T03:01:58.6452468Z Prioritized test from test file changes.
2022-11-23T03:01:58.6452820Z reordering tests for PR:
2022-11-23T03:01:58.6453204Z prioritized: []
2022-11-23T03:01:58.6453711Z the rest: ['distributed/fsdp/test_fsdp_misc.py']
2022-11-23T03:01:58.6453923Z 
2022-11-23T03:01:58.6454466Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:01:58.6455392Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:01:58.6460167Z parallel (file granularity) tests:
2022-11-23T03:01:58.6460455Z  
2022-11-23T03:01:58.6460688Z serial (file granularity) tests:
2022-11-23T03:01:58.6461010Z  distributed/fsdp/test_fsdp_misc.py
2022-11-23T03:02:00.9601725Z Ignoring disabled issues:  []
2022-11-23T03:02:00.9655290Z Ignoring disabled issues:  []
2022-11-23T03:02:01.4414931Z Running distributed/fsdp/test_fsdp_misc.py ... [2022-11-23 03:02:01.440757]
2022-11-23T03:02:01.4415704Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_misc.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:02:01.441205]
2022-11-23T03:02:05.6881726Z 
2022-11-23T03:02:05.6882431Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_misc
2022-11-23T03:02:05.6883696Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_misc (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_misc_hwbq517t)
2022-11-23T03:02:05.6884081Z 
2022-11-23T03:02:05.6884182Z Running tests...
2022-11-23T03:02:05.6884765Z ----------------------------------------------------------------------
2022-11-23T03:02:05.6885317Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_misc
2022-11-23T03:02:05.6885813Z   test_cpu_init_with_sync_module_states (__main__.TestFSDPMisc)
2022-11-23T03:02:05.6886260Z Tests that passing ``sync_module_states=True`` raises an error for ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:02:05.6887247Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.755s)
2022-11-23T03:02:05.6887739Z   test_device_id_auto_wrap (__main__.TestFSDPMisc)
2022-11-23T03:02:05.6888320Z Tests that ``auto_wrap_policy`` propagates ``device_id`` to all ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:05.6888831Z   test_fsdp_cpu_init_stays_on_cpu (__main__.TestFSDPMisc)
2022-11-23T03:02:05.6889543Z Tests that passing a CPU module to FSDP preserves that the wrapped ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:05.6890086Z   test_fsdp_device_id_cpu_offload (__main__.TestFSDPMisc)
2022-11-23T03:02:05.6890773Z Ensures that even if device_id is specified but we have ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:05.6891281Z   test_fsdp_device_id_use_index_False (__main__.TestFSDPMisc)
2022-11-23T03:02:05.6891930Z Tests the FSDP ``device_id`` argument: ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:02:05.6892430Z   test_fsdp_device_id_use_index_True (__main__.TestFSDPMisc)
2022-11-23T03:02:05.6893080Z Tests the FSDP ``device_id`` argument: ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:02:05.6893961Z   test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy_None (__main__.TestFSDPMisc) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:02:05.6894983Z   test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestFSDPMisc) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:02:05.6895994Z   test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy_None (__main__.TestFSDPMisc) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:02:05.6897522Z   test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestFSDPMisc) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:02:05.6898443Z   test_fsdp_namedtuple (__main__.TestFSDPMisc) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:05.6899411Z   test_fsdp_not_all_outputs_used_in_loss (__main__.TestFSDPMisc) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:02:05.6899976Z   test_fsdp_same_model_across_ranks (__main__.TestFSDPMisc)
2022-11-23T03:02:05.6900599Z FSDP broadcasts model from rank 0 to ensure it starts off with the same ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:02:05.6901147Z   test_module_device_mismatches_device_id (__main__.TestFSDPMisc)
2022-11-23T03:02:05.6901865Z Tests that specifying a ``device_id`` argument to FSDP for a GPU ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:05.6902377Z   test_multi_device_not_supported (__main__.TestFSDPMisc)
2022-11-23T03:02:05.6903084Z Tests that wrapping a multi-device module (i.e. with submodules on ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:05.6903590Z   test_no_params (__main__.TestFSDPMisc)
2022-11-23T03:02:05.6904328Z Test that device_id and cpu init work if module has no params ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:05.6904699Z 
2022-11-23T03:02:05.6904969Z ----------------------------------------------------------------------
2022-11-23T03:02:05.6905301Z Ran 16 tests in 1.778s
2022-11-23T03:02:05.6905462Z 
2022-11-23T03:02:05.6905574Z OK (skipped=16)
2022-11-23T03:02:05.6905731Z 
2022-11-23T03:02:05.6905837Z Generating XML reports...
2022-11-23T03:02:05.6906415Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_misc/TEST-TestFSDPMisc-20221123030203.xml
2022-11-23T03:02:05.6906754Z 
2022-11-23T03:02:05.6907071Z ##[endgroup]
2022-11-23T03:02:05.6907648Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_misc (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_misc_hwbq517t)
2022-11-23T03:02:05.6908001Z 
2022-11-23T03:02:06.0450710Z 
2022-11-23T03:02:06.0451385Z real	0m9.898s
2022-11-23T03:02:06.0451717Z user	0m16.687s
2022-11-23T03:02:06.0451965Z sys	0m14.309s
2022-11-23T03:02:06.0452224Z + for f in test/distributed/fsdp/*.py
2022-11-23T03:02:06.0452805Z + python test/run_test.py --verbose -i distributed/fsdp/test_fsdp_mixed_precision.py
2022-11-23T03:02:08.3851724Z Ignoring disabled issues:  []
2022-11-23T03:02:08.4384462Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:02:08.4385087Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:02:08.4385430Z Selected tests:
2022-11-23T03:02:08.4385752Z  distributed/fsdp/test_fsdp_mixed_precision.py
2022-11-23T03:02:08.4408959Z Prioritized test from test file changes.
2022-11-23T03:02:08.4409638Z reordering tests for PR:
2022-11-23T03:02:08.4409979Z prioritized: []
2022-11-23T03:02:08.4410561Z the rest: ['distributed/fsdp/test_fsdp_mixed_precision.py']
2022-11-23T03:02:08.4410794Z 
2022-11-23T03:02:08.4411308Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:02:08.4412189Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:02:08.4418340Z parallel (file granularity) tests:
2022-11-23T03:02:08.4418662Z  
2022-11-23T03:02:08.4418909Z serial (file granularity) tests:
2022-11-23T03:02:08.4419267Z  distributed/fsdp/test_fsdp_mixed_precision.py
2022-11-23T03:02:10.7486145Z Ignoring disabled issues:  []
2022-11-23T03:02:10.7486570Z Ignoring disabled issues:  []
2022-11-23T03:02:11.2688031Z Running distributed/fsdp/test_fsdp_mixed_precision.py ... [2022-11-23 03:02:11.268149]
2022-11-23T03:02:11.2689884Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_mixed_precision.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:02:11.268610]
2022-11-23T03:02:15.6119791Z 
2022-11-23T03:02:15.6120615Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_mixed_precision
2022-11-23T03:02:15.6121828Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_mixed_precision (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_mixed_precision_x_1zx0bu)
2022-11-23T03:02:15.6122251Z 
2022-11-23T03:02:15.6122400Z Running tests...
2022-11-23T03:02:15.6122955Z ----------------------------------------------------------------------
2022-11-23T03:02:15.6123532Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision
2022-11-23T03:02:15.6124274Z   test_grads_reduced_precision (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.016s)
2022-11-23T03:02:15.6125713Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6126840Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6127961Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6129075Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6130206Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6131313Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6132442Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6133546Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6134648Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6135691Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6137230Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6138450Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6139476Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6140550Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6141526Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6142679Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6143766Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6144828Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6145911Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6146975Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6148034Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6149100Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6150182Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6151243Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6152345Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6153503Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6154624Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6155726Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6156843Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6157944Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6159121Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6160219Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6161325Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6162398Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6163517Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6164564Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6165664Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6166743Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6167848Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6168900Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6169959Z   test_mixed_precision_no_reshard_after_forward (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6170583Z   test_mixed_precision_resnet (__main__.TestFSDPMixedPrecisionSharded)
2022-11-23T03:02:15.6171044Z End to end test to ensure mixed precision + auto_wrap works ... skip: no torchvision (0.001s)
2022-11-23T03:02:15.6171826Z   test_mp_batchnorm_convert_sync_bn_False (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:02:15.6172774Z   test_mp_batchnorm_convert_sync_bn_True (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:02:15.6173695Z   test_mp_embedding_default (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6174628Z   test_mp_embedding_only_params_and_bufs (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6175637Z   test_mp_embedding_params_and_reduce_diff (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6176544Z   test_mp_embedding_reduce (__main__.TestFSDPMixedPrecisionSharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6177689Z   test_grads_reduced_precision (__main__.TestFSDPMixedPrecisionUnsharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6178642Z   test_mixed_precision_e2e_full_shard (__main__.TestFSDPMixedPrecisionUnsharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6179618Z   test_mixed_precision_no_reshard_after_forward (__main__.TestFSDPMixedPrecisionUnsharded) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:15.6180048Z 
2022-11-23T03:02:15.6180315Z ----------------------------------------------------------------------
2022-11-23T03:02:15.6180629Z Ran 52 tests in 0.063s
2022-11-23T03:02:15.6180794Z 
2022-11-23T03:02:15.6180904Z OK (skipped=52)
2022-11-23T03:02:15.6181063Z 
2022-11-23T03:02:15.6181190Z Generating XML reports...
2022-11-23T03:02:15.6181842Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionSharded-20221123030215.xml
2022-11-23T03:02:15.6182742Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionUnsharded-20221123030215.xml
2022-11-23T03:02:15.6183242Z 
2022-11-23T03:02:15.6183587Z ##[endgroup]
2022-11-23T03:02:15.6184232Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_mixed_precision (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_mixed_precision_x_1zx0bu)
2022-11-23T03:02:15.6184617Z 
2022-11-23T03:02:15.9437162Z 
2022-11-23T03:02:15.9438339Z real	0m9.899s
2022-11-23T03:02:15.9438563Z user	0m17.174s
2022-11-23T03:02:15.9438809Z sys	0m13.113s
2022-11-23T03:02:15.9439103Z + for f in test/distributed/fsdp/*.py
2022-11-23T03:02:15.9439740Z + python test/run_test.py --verbose -i distributed/fsdp/test_fsdp_multiple_forward.py
2022-11-23T03:02:18.2661997Z Ignoring disabled issues:  []
2022-11-23T03:02:18.3196522Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:02:18.3197086Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:02:18.3197691Z Selected tests:
2022-11-23T03:02:18.3198037Z  distributed/fsdp/test_fsdp_multiple_forward.py
2022-11-23T03:02:18.3226969Z Prioritized test from test file changes.
2022-11-23T03:02:18.3227840Z reordering tests for PR:
2022-11-23T03:02:18.3228278Z prioritized: []
2022-11-23T03:02:18.3228905Z the rest: ['distributed/fsdp/test_fsdp_multiple_forward.py']
2022-11-23T03:02:18.3229080Z 
2022-11-23T03:02:18.3229613Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:02:18.3230556Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:02:18.3235947Z parallel (file granularity) tests:
2022-11-23T03:02:18.3236669Z  
2022-11-23T03:02:18.3237075Z serial (file granularity) tests:
2022-11-23T03:02:18.3237730Z  distributed/fsdp/test_fsdp_multiple_forward.py
2022-11-23T03:02:20.6400540Z Ignoring disabled issues:  []
2022-11-23T03:02:20.6511573Z Ignoring disabled issues:  []
2022-11-23T03:02:21.0428254Z Running distributed/fsdp/test_fsdp_multiple_forward.py ... [2022-11-23 03:02:21.042135]
2022-11-23T03:02:21.0429236Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_forward.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:02:21.042601]
2022-11-23T03:02:25.1826070Z 
2022-11-23T03:02:25.1826885Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_multiple_forward
2022-11-23T03:02:25.1827971Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_forward (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_multiple_forward_c5d5e7ku)
2022-11-23T03:02:25.1828374Z 
2022-11-23T03:02:25.1828498Z Running tests...
2022-11-23T03:02:25.1828957Z ----------------------------------------------------------------------
2022-11-23T03:02:25.1829558Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward
2022-11-23T03:02:25.1830093Z   test_multi_forward (__main__.TestMultiForward) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:02:25.1830741Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.732s)
2022-11-23T03:02:25.1831040Z 
2022-11-23T03:02:25.1831304Z ----------------------------------------------------------------------
2022-11-23T03:02:25.1831615Z Ran 1 test in 1.733s
2022-11-23T03:02:25.1831779Z 
2022-11-23T03:02:25.1831886Z OK (skipped=1)
2022-11-23T03:02:25.1832039Z 
2022-11-23T03:02:25.1832165Z Generating XML reports...
2022-11-23T03:02:25.1832765Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward/TEST-TestMultiForward-20221123030223.xml
2022-11-23T03:02:25.1833127Z 
2022-11-23T03:02:25.1833436Z ##[endgroup]
2022-11-23T03:02:25.1834079Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_forward (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_multiple_forward_c5d5e7ku)
2022-11-23T03:02:25.1834465Z 
2022-11-23T03:02:25.5158554Z 
2022-11-23T03:02:25.5159031Z real	0m9.572s
2022-11-23T03:02:25.5159365Z user	0m17.029s
2022-11-23T03:02:25.5159630Z sys	0m14.310s
2022-11-23T03:02:25.5159908Z + for f in test/distributed/fsdp/*.py
2022-11-23T03:02:25.5160579Z + python test/run_test.py --verbose -i distributed/fsdp/test_fsdp_multiple_wrapping.py
2022-11-23T03:02:27.9178055Z Ignoring disabled issues:  []
2022-11-23T03:02:27.9715666Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:02:27.9716275Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:02:27.9716634Z Selected tests:
2022-11-23T03:02:27.9716948Z  distributed/fsdp/test_fsdp_multiple_wrapping.py
2022-11-23T03:02:27.9741278Z Prioritized test from test file changes.
2022-11-23T03:02:27.9741663Z reordering tests for PR:
2022-11-23T03:02:27.9741956Z prioritized: []
2022-11-23T03:02:27.9742487Z the rest: ['distributed/fsdp/test_fsdp_multiple_wrapping.py']
2022-11-23T03:02:27.9742728Z 
2022-11-23T03:02:27.9743283Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:02:27.9744231Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:02:27.9751651Z parallel (file granularity) tests:
2022-11-23T03:02:27.9751955Z  
2022-11-23T03:02:27.9752184Z serial (file granularity) tests:
2022-11-23T03:02:27.9752525Z  distributed/fsdp/test_fsdp_multiple_wrapping.py
2022-11-23T03:02:30.2899329Z Ignoring disabled issues:  []
2022-11-23T03:02:30.3057868Z Ignoring disabled issues:  []
2022-11-23T03:02:30.7559804Z Running distributed/fsdp/test_fsdp_multiple_wrapping.py ... [2022-11-23 03:02:30.755453]
2022-11-23T03:02:30.7561058Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_wrapping.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:02:30.755847]
2022-11-23T03:02:34.9962282Z 
2022-11-23T03:02:34.9963121Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_multiple_wrapping
2022-11-23T03:02:34.9964211Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_wrapping (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_multiple_wrapping_12953xmn)
2022-11-23T03:02:34.9964613Z 
2022-11-23T03:02:34.9964709Z Running tests...
2022-11-23T03:02:34.9965256Z ----------------------------------------------------------------------
2022-11-23T03:02:34.9965855Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping
2022-11-23T03:02:34.9966353Z   test_multiple_wrapping (__main__.TestMultipleWrapping)
2022-11-23T03:02:34.9966816Z This test simulates wrapping the module after training to run inference. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:02:34.9967502Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.782s)
2022-11-23T03:02:34.9967807Z 
2022-11-23T03:02:34.9968077Z ----------------------------------------------------------------------
2022-11-23T03:02:34.9968387Z Ran 1 test in 1.782s
2022-11-23T03:02:34.9968553Z 
2022-11-23T03:02:34.9968665Z OK (skipped=1)
2022-11-23T03:02:34.9968822Z 
2022-11-23T03:02:34.9968950Z Generating XML reports...
2022-11-23T03:02:34.9969589Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20221123030232.xml
2022-11-23T03:02:34.9969979Z 
2022-11-23T03:02:34.9970266Z ##[endgroup]
2022-11-23T03:02:34.9970923Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_wrapping (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_multiple_wrapping_12953xmn)
2022-11-23T03:02:34.9971316Z 
2022-11-23T03:02:35.3795898Z 
2022-11-23T03:02:35.3796380Z real	0m9.864s
2022-11-23T03:02:35.3796655Z user	0m18.141s
2022-11-23T03:02:35.3796941Z sys	0m15.731s
2022-11-23T03:02:35.3797230Z + for f in test/distributed/fsdp/*.py
2022-11-23T03:02:35.3797860Z + python test/run_test.py --verbose -i distributed/fsdp/test_fsdp_optim_state.py
2022-11-23T03:02:37.7172107Z Ignoring disabled issues:  []
2022-11-23T03:02:37.7713121Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:02:37.7713759Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:02:37.7714104Z Selected tests:
2022-11-23T03:02:37.7714409Z  distributed/fsdp/test_fsdp_optim_state.py
2022-11-23T03:02:37.7739724Z Prioritized test from test file changes.
2022-11-23T03:02:37.7740368Z reordering tests for PR:
2022-11-23T03:02:37.7740716Z prioritized: []
2022-11-23T03:02:37.7741215Z the rest: ['distributed/fsdp/test_fsdp_optim_state.py']
2022-11-23T03:02:37.7741465Z 
2022-11-23T03:02:37.7742115Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:02:37.7743143Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:02:37.7747619Z parallel (file granularity) tests:
2022-11-23T03:02:37.7747896Z  
2022-11-23T03:02:37.7748203Z serial (file granularity) tests:
2022-11-23T03:02:37.7748523Z  distributed/fsdp/test_fsdp_optim_state.py
2022-11-23T03:02:40.0830733Z Ignoring disabled issues:  []
2022-11-23T03:02:40.1510797Z Ignoring disabled issues:  []
2022-11-23T03:02:40.5359187Z Running distributed/fsdp/test_fsdp_optim_state.py ... [2022-11-23 03:02:40.535249]
2022-11-23T03:02:40.5360089Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_optim_state.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:02:40.535721]
2022-11-23T03:02:44.7807787Z 
2022-11-23T03:02:44.7808291Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_optim_state
2022-11-23T03:02:44.7809365Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_optim_state (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_optim_state_t3j84mic)
2022-11-23T03:02:44.7809786Z 
2022-11-23T03:02:44.7809906Z Running tests...
2022-11-23T03:02:44.7810449Z ----------------------------------------------------------------------
2022-11-23T03:02:44.7811010Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state
2022-11-23T03:02:44.7811501Z   test_flatten_sharded_optim_state_dict_nested (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7812107Z Tests :meth:`flatten_sharded_optim_state_dict` for an FSDP-root ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:02:44.7812819Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.758s)
2022-11-23T03:02:44.7813301Z   test_flatten_sharded_optim_state_dict_transformer (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7814045Z Tests :meth:`flatten_sharded_optim_state_dict` for an FSDP-root ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7814587Z   test_full_optim_state_dict_keys (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7815257Z Tests that the parameter keys returned by ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7815771Z   test_full_optim_state_dict_nested_invalid (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7816495Z Tests that :meth:`full_optim_state_dict` raises an error when ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7817281Z   test_optim_input_warning (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7818001Z Tests that passing the ``optim_input`` argument into optimizer state ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7818712Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7819579Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7820608Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7821509Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7822211Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7823047Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7823751Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7824614Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7825436Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7826287Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7826985Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7827843Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7828538Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7829392Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7830075Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7830934Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7831640Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7832503Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7833212Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7834058Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7834763Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7835684Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7836398Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7837260Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7837938Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7838804Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7839504Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7840431Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7841126Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7841956Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7842660Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7843517Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7844174Z   test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7844951Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7845596Z   test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7846390Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7847045Z   test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7847845Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7848468Z   test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7849265Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7849807Z   test_rekey_optim_state_dict_to_names (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7850560Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7851153Z   test_save_load_without_0th_param_state_state_dict_type_StateDictType_FULL_STATE_DICT (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7851954Z Tests saving and loading an optim state dict for Adam optimizer (i.e. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:02:44.7852585Z   test_save_load_without_0th_param_state_state_dict_type_StateDictType_SHARDED_STATE_DICT (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7853372Z Tests saving and loading an optim state dict for Adam optimizer (i.e. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:02:44.7853934Z   test_scatter_full_optim_state_dict_nested_halve_world_size (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7854721Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7855390Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7856266Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7857281Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7858093Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7858755Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7859564Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7860220Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7861032Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7861669Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7862602Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7863262Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7864079Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7864715Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7865527Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7866176Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7867073Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7867654Z   test_scatter_full_optim_state_dict_transformer (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7868366Z Tests :meth:`scatter_full_optim_state_dict` for an FSDP-root ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7868936Z   test_shard_full_optim_state_dict_nested_halve_world_size (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7869686Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7870349Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7871149Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7871884Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7872699Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7873360Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7874151Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7874812Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7875625Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7876278Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7877081Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7877718Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7878530Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7879198Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7880016Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7880650Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7881463Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7882019Z   test_shard_full_optim_state_dict_transformer (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7882788Z Tests :meth:`shard_full_optim_state_dict` for an FSDP-root ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7883438Z   test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_FULL_STATE_DICT_add_to_fsdp_module_False (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7884264Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7884930Z   test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_FULL_STATE_DICT_add_to_fsdp_module_True (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7885750Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7886420Z   test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_SHARDED_STATE_DICT_add_to_fsdp_module_False (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7887226Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7887953Z   test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_SHARDED_STATE_DICT_add_to_fsdp_module_True (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7888779Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7889309Z   test_use_orig_params_error (__main__.TestFSDPOptimState)
2022-11-23T03:02:44.7889997Z Tests that the optimizer state checkpointing APIs raise an error ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:02:44.7890381Z 
2022-11-23T03:02:44.7890660Z ----------------------------------------------------------------------
2022-11-23T03:02:44.7890996Z Ran 53 tests in 1.816s
2022-11-23T03:02:44.7891164Z 
2022-11-23T03:02:44.7891274Z OK (skipped=53)
2022-11-23T03:02:44.7891411Z 
2022-11-23T03:02:44.7891538Z Generating XML reports...
2022-11-23T03:02:44.7892151Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state/TEST-TestFSDPOptimState-20221123030242.xml
2022-11-23T03:02:44.7892517Z 
2022-11-23T03:02:44.7892889Z ##[endgroup]
2022-11-23T03:02:44.7893493Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_optim_state (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_optim_state_t3j84mic)
2022-11-23T03:02:44.7893861Z 
2022-11-23T03:02:45.1695584Z 
2022-11-23T03:02:45.1696089Z real	0m9.790s
2022-11-23T03:02:45.1696397Z user	0m17.855s
2022-11-23T03:02:45.1696621Z sys	0m16.638s
2022-11-23T03:02:45.1697241Z + for f in test/distributed/fsdp/*.py
2022-11-23T03:02:45.1697879Z + python test/run_test.py --verbose -i distributed/fsdp/test_fsdp_overlap.py
2022-11-23T03:02:47.5261309Z Ignoring disabled issues:  []
2022-11-23T03:02:47.5796440Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:02:47.5797026Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:02:47.5797388Z Selected tests:
2022-11-23T03:02:47.5797657Z  distributed/fsdp/test_fsdp_overlap.py
2022-11-23T03:02:47.5822874Z Prioritized test from test file changes.
2022-11-23T03:02:47.5823208Z reordering tests for PR:
2022-11-23T03:02:47.5823484Z prioritized: []
2022-11-23T03:02:47.5823984Z the rest: ['distributed/fsdp/test_fsdp_overlap.py']
2022-11-23T03:02:47.5824201Z 
2022-11-23T03:02:47.5824734Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:02:47.5825984Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:02:47.5830156Z parallel (file granularity) tests:
2022-11-23T03:02:47.5830441Z  
2022-11-23T03:02:47.5830689Z serial (file granularity) tests:
2022-11-23T03:02:47.5830996Z  distributed/fsdp/test_fsdp_overlap.py
2022-11-23T03:02:49.8904526Z Ignoring disabled issues:  []
2022-11-23T03:02:49.8914046Z Ignoring disabled issues:  []
2022-11-23T03:02:50.4242777Z Running distributed/fsdp/test_fsdp_overlap.py ... [2022-11-23 03:02:50.423675]
2022-11-23T03:02:50.4244048Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_overlap.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:02:50.424151]
2022-11-23T03:06:23.8411886Z 
2022-11-23T03:06:23.8412612Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_overlap
2022-11-23T03:06:23.8416481Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_overlap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_overlap_ccye88j5)
2022-11-23T03:06:23.8418044Z 
2022-11-23T03:06:23.8418236Z Running tests...
2022-11-23T03:06:23.8419047Z ----------------------------------------------------------------------
2022-11-23T03:06:23.8419628Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap
2022-11-23T03:06:23.8420175Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:06:23.8420850Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.722s)
2022-11-23T03:06:23.8421460Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 128574
2022-11-23T03:06:23.8422022Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 128575
2022-11-23T03:06:23.8422913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.8423547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.8424673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.8425618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.8426579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.8427469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.8428059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.8428527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.8428977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.8429472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.8430140Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.8430838Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.8431342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.8431826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.8433954Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.8435420Z   warnings.warn(
2022-11-23T03:06:23.8437591Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.8438966Z   warnings.warn(
2022-11-23T03:06:23.8439535Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.8440526Z Traceback (most recent call last):
2022-11-23T03:06:23.8441552Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8442234Z     getattr(self, test_name)()
2022-11-23T03:06:23.8443284Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8443949Z     fn()
2022-11-23T03:06:23.8444834Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8445528Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8446217Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8446890Z     self._dist_train()
2022-11-23T03:06:23.8447579Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8448455Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8449221Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8449939Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8450568Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8451187Z     raise self.failureException(msg)
2022-11-23T03:06:23.8451737Z AssertionError: False is not true
2022-11-23T03:06:23.8452249Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.8452915Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.8453563Z Traceback (most recent call last):
2022-11-23T03:06:23.8454527Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8455193Z     getattr(self, test_name)()
2022-11-23T03:06:23.8456143Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8457169Z     fn()
2022-11-23T03:06:23.8458123Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8458839Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8459564Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8460233Z     self._dist_train()
2022-11-23T03:06:23.8460908Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8461764Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8462486Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8463160Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8463819Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8464522Z     raise self.failureException(msg)
2022-11-23T03:06:23.8465065Z AssertionError: False is not true
2022-11-23T03:06:23.8465752Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.8466262Z dist init r=1, world=2
2022-11-23T03:06:23.8466679Z dist init r=0, world=2
2022-11-23T03:06:23.8467266Z Process 1 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.8467844Z ERROR (4.115s)
2022-11-23T03:06:23.8468799Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 50
2022-11-23T03:06:23.8469597Z Traceback (most recent call last):
2022-11-23T03:06:23.8470598Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.8471309Z     self._join_processes(fn)
2022-11-23T03:06:23.8472252Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.8473027Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.8474038Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.8474767Z     raise RuntimeError(error)
2022-11-23T03:06:23.8475469Z RuntimeError: Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.8476123Z Traceback (most recent call last):
2022-11-23T03:06:23.8477087Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8477811Z     getattr(self, test_name)()
2022-11-23T03:06:23.8478769Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8479427Z     fn()
2022-11-23T03:06:23.8480347Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8481061Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8481837Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8482474Z     self._dist_train()
2022-11-23T03:06:23.8483170Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8484042Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8484806Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8485546Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8486143Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8486803Z     raise self.failureException(msg)
2022-11-23T03:06:23.8487367Z AssertionError: False is not true
2022-11-23T03:06:23.8487713Z 
2022-11-23T03:06:23.8487736Z 
2022-11-23T03:06:23.8487768Z 
2022-11-23T03:06:23.8488144Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 128725
2022-11-23T03:06:23.8488984Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 128726
2022-11-23T03:06:23.8490106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.8490928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.8491963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.8492856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.8493901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.8494723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.8495729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.8496615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.8497903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.8498859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.8500058Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.8501305Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.8502272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.8503136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.8505464Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.8506991Z   warnings.warn(
2022-11-23T03:06:23.8509327Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.8510150Z   warnings.warn(
2022-11-23T03:06:23.8510497Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.8510841Z Traceback (most recent call last):
2022-11-23T03:06:23.8511390Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8511797Z     getattr(self, test_name)()
2022-11-23T03:06:23.8512301Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8512686Z     fn()
2022-11-23T03:06:23.8513178Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8513565Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8513951Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8514329Z     self._dist_train()
2022-11-23T03:06:23.8514713Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8515190Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8515612Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8515994Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8516340Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8516695Z     raise self.failureException(msg)
2022-11-23T03:06:23.8516999Z AssertionError: False is not true
2022-11-23T03:06:23.8517296Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.8517645Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.8517993Z Traceback (most recent call last):
2022-11-23T03:06:23.8518524Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8518902Z     getattr(self, test_name)()
2022-11-23T03:06:23.8519415Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8519775Z     fn()
2022-11-23T03:06:23.8520302Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8520708Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8521107Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8521477Z     self._dist_train()
2022-11-23T03:06:23.8521834Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8522322Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8522739Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8523103Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8523465Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8523818Z     raise self.failureException(msg)
2022-11-23T03:06:23.8524105Z AssertionError: False is not true
2022-11-23T03:06:23.8524403Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.8524677Z dist init r=0, world=2
2022-11-23T03:06:23.8524986Z dist init r=1, world=2
2022-11-23T03:06:23.8525310Z Process 1 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.8525636Z ERROR (4.113s)
2022-11-23T03:06:23.8526171Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 49
2022-11-23T03:06:23.8526579Z Traceback (most recent call last):
2022-11-23T03:06:23.8527106Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.8527500Z     self._join_processes(fn)
2022-11-23T03:06:23.8528010Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.8528431Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.8528990Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.8529399Z     raise RuntimeError(error)
2022-11-23T03:06:23.8529722Z RuntimeError: Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.8530071Z Traceback (most recent call last):
2022-11-23T03:06:23.8530615Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8530992Z     getattr(self, test_name)()
2022-11-23T03:06:23.8531523Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8531899Z     fn()
2022-11-23T03:06:23.8532367Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8532761Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8533162Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8533529Z     self._dist_train()
2022-11-23T03:06:23.8533890Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8534375Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8534793Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8535158Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8535534Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8535897Z     raise self.failureException(msg)
2022-11-23T03:06:23.8536212Z AssertionError: False is not true
2022-11-23T03:06:23.8536378Z 
2022-11-23T03:06:23.8536397Z 
2022-11-23T03:06:23.8536416Z 
2022-11-23T03:06:23.8536639Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 128876
2022-11-23T03:06:23.8537383Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 128877
2022-11-23T03:06:23.8538027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.8538469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.8539056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.8539532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.8540122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.8540551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.8541136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.8541607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.8542080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.8542660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.8543318Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.8544006Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.8544518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.8545160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.8546438Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.8547213Z   warnings.warn(
2022-11-23T03:06:23.8548369Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.8549136Z   warnings.warn(
2022-11-23T03:06:23.8549459Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.8549814Z Traceback (most recent call last):
2022-11-23T03:06:23.8550348Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8550726Z     getattr(self, test_name)()
2022-11-23T03:06:23.8551238Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8551600Z     fn()
2022-11-23T03:06:23.8552060Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8552452Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8552849Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8553218Z     self._dist_train()
2022-11-23T03:06:23.8553578Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8554062Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8554549Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8554926Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8555292Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8555645Z     raise self.failureException(msg)
2022-11-23T03:06:23.8555935Z AssertionError: False is not true
2022-11-23T03:06:23.8556229Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.8556596Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.8556946Z Traceback (most recent call last):
2022-11-23T03:06:23.8557464Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8557858Z     getattr(self, test_name)()
2022-11-23T03:06:23.8558375Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8558719Z     fn()
2022-11-23T03:06:23.8559203Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8559651Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8560038Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8560412Z     self._dist_train()
2022-11-23T03:06:23.8560789Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8561360Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8561801Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8562187Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8562552Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8562890Z     raise self.failureException(msg)
2022-11-23T03:06:23.8563204Z AssertionError: False is not true
2022-11-23T03:06:23.8563501Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.8563765Z dist init r=0, world=2
2022-11-23T03:06:23.8564011Z dist init r=1, world=2
2022-11-23T03:06:23.8564357Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.8564692Z ERROR (4.113s)
2022-11-23T03:06:23.8565215Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 48
2022-11-23T03:06:23.8565652Z Traceback (most recent call last):
2022-11-23T03:06:23.8566189Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.8566567Z     self._join_processes(fn)
2022-11-23T03:06:23.8567107Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.8567535Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.8568080Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.8568503Z     raise RuntimeError(error)
2022-11-23T03:06:23.8568852Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.8569202Z Traceback (most recent call last):
2022-11-23T03:06:23.8569720Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8570122Z     getattr(self, test_name)()
2022-11-23T03:06:23.8570647Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8570999Z     fn()
2022-11-23T03:06:23.8571485Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8571880Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8572344Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8572711Z     self._dist_train()
2022-11-23T03:06:23.8595395Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8595933Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8596368Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8596754Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8597379Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8597725Z     raise self.failureException(msg)
2022-11-23T03:06:23.8598039Z AssertionError: False is not true
2022-11-23T03:06:23.8598229Z 
2022-11-23T03:06:23.8598248Z 
2022-11-23T03:06:23.8598266Z 
2022-11-23T03:06:23.8598493Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 129027
2022-11-23T03:06:23.8598945Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 129028
2022-11-23T03:06:23.8599582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.8600194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.8600778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.8601237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.8601815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.8602260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.8602813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.8603276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.8603722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.8604227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.8604865Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.8605551Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.8606071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.8606541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.8607791Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.8608578Z   warnings.warn(
2022-11-23T03:06:23.8609722Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.8610536Z   warnings.warn(
2022-11-23T03:06:23.8610874Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.8611213Z Traceback (most recent call last):
2022-11-23T03:06:23.8611827Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8612218Z     getattr(self, test_name)()
2022-11-23T03:06:23.8612710Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8613063Z     fn()
2022-11-23T03:06:23.8613547Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8613931Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8614320Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8614700Z     self._dist_train()
2022-11-23T03:06:23.8615074Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8615538Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8615962Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8616394Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8616982Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8617340Z     raise self.failureException(msg)
2022-11-23T03:06:23.8617640Z AssertionError: False is not true
2022-11-23T03:06:23.8617939Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.8618203Z dist init r=1, world=2
2022-11-23T03:06:23.8618543Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.8618894Z Traceback (most recent call last):
2022-11-23T03:06:23.8619423Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8619818Z     getattr(self, test_name)()
2022-11-23T03:06:23.8620343Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8620690Z     fn()
2022-11-23T03:06:23.8621177Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8621569Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8621975Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8622329Z     self._dist_train()
2022-11-23T03:06:23.8622704Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8623191Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8623588Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8623965Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8624324Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8624667Z     raise self.failureException(msg)
2022-11-23T03:06:23.8624980Z AssertionError: False is not true
2022-11-23T03:06:23.8625279Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.8625555Z dist init r=0, world=2
2022-11-23T03:06:23.8625876Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.8626201Z ERROR (4.013s)
2022-11-23T03:06:23.8626732Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 47
2022-11-23T03:06:23.8627141Z Traceback (most recent call last):
2022-11-23T03:06:23.8627671Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.8628062Z     self._join_processes(fn)
2022-11-23T03:06:23.8628570Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.8628994Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.8629631Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.8630061Z     raise RuntimeError(error)
2022-11-23T03:06:23.8630383Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.8630728Z Traceback (most recent call last):
2022-11-23T03:06:23.8631261Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8631640Z     getattr(self, test_name)()
2022-11-23T03:06:23.8632159Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8632525Z     fn()
2022-11-23T03:06:23.8633008Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8633379Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8633785Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8634156Z     self._dist_train()
2022-11-23T03:06:23.8634592Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8635085Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8635521Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8635886Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8636251Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8636623Z     raise self.failureException(msg)
2022-11-23T03:06:23.8636944Z AssertionError: False is not true
2022-11-23T03:06:23.8637108Z 
2022-11-23T03:06:23.8637127Z 
2022-11-23T03:06:23.8637167Z 
2022-11-23T03:06:23.8637374Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 129178
2022-11-23T03:06:23.8637845Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 129179
2022-11-23T03:06:23.8638462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.8638932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.8639491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.8639972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.8640560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.8641026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.8641604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.8642075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.8642540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.8643032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.8643701Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.8644397Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.8644934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.8645394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.8646716Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.8647524Z   warnings.warn(
2022-11-23T03:06:23.8648686Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.8649469Z   warnings.warn(
2022-11-23T03:06:23.8649800Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.8650165Z Traceback (most recent call last):
2022-11-23T03:06:23.8650862Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8651282Z     getattr(self, test_name)()
2022-11-23T03:06:23.8651794Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8652241Z     fn()
2022-11-23T03:06:23.8652698Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8653096Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8653499Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8653866Z     self._dist_train()
2022-11-23T03:06:23.8654218Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8654695Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8655114Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8655475Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8655846Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8656208Z     raise self.failureException(msg)
2022-11-23T03:06:23.8656525Z AssertionError: False is not true
2022-11-23T03:06:23.8657346Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.8657653Z dist init r=0, world=2
2022-11-23T03:06:23.8658011Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.8658353Z Traceback (most recent call last):
2022-11-23T03:06:23.8658913Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8659325Z     getattr(self, test_name)()
2022-11-23T03:06:23.8659832Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8660211Z     fn()
2022-11-23T03:06:23.8660705Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8661111Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8661500Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8661935Z     self._dist_train()
2022-11-23T03:06:23.8662333Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8662818Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8663248Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8663642Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8663990Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8664591Z     raise self.failureException(msg)
2022-11-23T03:06:23.8664919Z AssertionError: False is not true
2022-11-23T03:06:23.8665220Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.8665476Z dist init r=1, world=2
2022-11-23T03:06:23.8665812Z Process 1 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.8666134Z ERROR (4.113s)
2022-11-23T03:06:23.8666641Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 46
2022-11-23T03:06:23.8667067Z Traceback (most recent call last):
2022-11-23T03:06:23.8667591Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.8667960Z     self._join_processes(fn)
2022-11-23T03:06:23.8668488Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.8668905Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.8669455Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.8669924Z     raise RuntimeError(error)
2022-11-23T03:06:23.8670264Z RuntimeError: Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.8670611Z Traceback (most recent call last):
2022-11-23T03:06:23.8671113Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8671508Z     getattr(self, test_name)()
2022-11-23T03:06:23.8672014Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8672528Z     fn()
2022-11-23T03:06:23.8673021Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8673423Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8673835Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8674198Z     self._dist_train()
2022-11-23T03:06:23.8674579Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8675078Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8675480Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8676019Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8676389Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8676740Z     raise self.failureException(msg)
2022-11-23T03:06:23.8677024Z AssertionError: False is not true
2022-11-23T03:06:23.8677375Z 
2022-11-23T03:06:23.8677394Z 
2022-11-23T03:06:23.8677413Z 
2022-11-23T03:06:23.8677637Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 129329
2022-11-23T03:06:23.8678113Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 129330
2022-11-23T03:06:23.8678717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.8679181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.8679778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.8680403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.8680950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.8681389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.8682125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.8682656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.8683109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.8683619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.8684292Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.8685117Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.8685639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.8686107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.8687513Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.8688363Z   warnings.warn(
2022-11-23T03:06:23.8689499Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.8690269Z   warnings.warn(
2022-11-23T03:06:23.8690607Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.8690969Z Traceback (most recent call last):
2022-11-23T03:06:23.8691492Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8691895Z     getattr(self, test_name)()
2022-11-23T03:06:23.8692569Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8692914Z     fn()
2022-11-23T03:06:23.8693564Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8693959Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8694368Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8694726Z     self._dist_train()
2022-11-23T03:06:23.8695112Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8695619Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8696021Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8696572Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8697425Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8697777Z     raise self.failureException(msg)
2022-11-23T03:06:23.8698105Z AssertionError: False is not true
2022-11-23T03:06:23.8698422Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.8698802Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.8699145Z Traceback (most recent call last):
2022-11-23T03:06:23.8699698Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8700112Z     getattr(self, test_name)()
2022-11-23T03:06:23.8700617Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8701069Z     fn()
2022-11-23T03:06:23.8701584Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8701968Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8702378Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8702757Z     self._dist_train()
2022-11-23T03:06:23.8703144Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8703768Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8704193Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8704572Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8704910Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8705274Z     raise self.failureException(msg)
2022-11-23T03:06:23.8705588Z AssertionError: False is not true
2022-11-23T03:06:23.8705862Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.8706220Z dist init r=0, world=2
2022-11-23T03:06:23.8706477Z dist init r=1, world=2
2022-11-23T03:06:23.8706824Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.8707127Z ERROR (4.013s)
2022-11-23T03:06:23.8707837Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 45
2022-11-23T03:06:23.8708330Z Traceback (most recent call last):
2022-11-23T03:06:23.8708873Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.8709284Z     self._join_processes(fn)
2022-11-23T03:06:23.8709830Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.8710292Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.8711190Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.8711617Z     raise RuntimeError(error)
2022-11-23T03:06:23.8711972Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.8713723Z Traceback (most recent call last):
2022-11-23T03:06:23.8714272Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8714833Z     getattr(self, test_name)()
2022-11-23T03:06:23.8715317Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8715682Z     fn()
2022-11-23T03:06:23.8716160Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8716539Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8716917Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8717287Z     self._dist_train()
2022-11-23T03:06:23.8717664Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8718122Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8718537Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8718922Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8719265Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8719807Z     raise self.failureException(msg)
2022-11-23T03:06:23.8720121Z AssertionError: False is not true
2022-11-23T03:06:23.8720306Z 
2022-11-23T03:06:23.8720325Z 
2022-11-23T03:06:23.8720343Z 
2022-11-23T03:06:23.8720569Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 129480
2022-11-23T03:06:23.8721116Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 129481
2022-11-23T03:06:23.8721764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.8722233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.8722820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.8723276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.8724015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.8724453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.8724989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.8725453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.8725902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.8726476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.8727102Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.8728079Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.8728614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.8729103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.8730356Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.8731290Z   warnings.warn(
2022-11-23T03:06:23.8732699Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.8733483Z   warnings.warn(
2022-11-23T03:06:23.8733837Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.8734184Z Traceback (most recent call last):
2022-11-23T03:06:23.8734729Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8735138Z     getattr(self, test_name)()
2022-11-23T03:06:23.8735967Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8736349Z     fn()
2022-11-23T03:06:23.8737071Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8737481Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8737869Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8738250Z     self._dist_train()
2022-11-23T03:06:23.8738644Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8739200Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8739643Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8740037Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8740385Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8740744Z     raise self.failureException(msg)
2022-11-23T03:06:23.8741064Z AssertionError: False is not true
2022-11-23T03:06:23.8741370Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.8741727Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.8742254Z Traceback (most recent call last):
2022-11-23T03:06:23.8742785Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8743153Z     getattr(self, test_name)()
2022-11-23T03:06:23.8743666Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8744033Z     fn()
2022-11-23T03:06:23.8744483Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8745007Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8745407Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8745775Z     self._dist_train()
2022-11-23T03:06:23.8746125Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8746611Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8747025Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8747559Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8747930Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8748299Z     raise self.failureException(msg)
2022-11-23T03:06:23.8748593Z AssertionError: False is not true
2022-11-23T03:06:23.8748905Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.8749190Z dist init r=1, world=2
2022-11-23T03:06:23.8749450Z dist init r=0, world=2
2022-11-23T03:06:23.8749776Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.8750113Z ERROR (4.113s)
2022-11-23T03:06:23.8750806Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 44
2022-11-23T03:06:23.8751209Z Traceback (most recent call last):
2022-11-23T03:06:23.8751733Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.8752125Z     self._join_processes(fn)
2022-11-23T03:06:23.8752624Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.8753052Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.8753599Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.8754002Z     raise RuntimeError(error)
2022-11-23T03:06:23.8754321Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.8754659Z Traceback (most recent call last):
2022-11-23T03:06:23.8755180Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8755544Z     getattr(self, test_name)()
2022-11-23T03:06:23.8756053Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8756414Z     fn()
2022-11-23T03:06:23.8756863Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8757251Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8757887Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8758289Z     self._dist_train()
2022-11-23T03:06:23.8758657Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8759163Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8759593Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8759962Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8760338Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8760706Z     raise self.failureException(msg)
2022-11-23T03:06:23.8761024Z AssertionError: False is not true
2022-11-23T03:06:23.8761190Z 
2022-11-23T03:06:23.8761209Z 
2022-11-23T03:06:23.8761228Z 
2022-11-23T03:06:23.8761459Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 129631
2022-11-23T03:06:23.8761981Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 129632
2022-11-23T03:06:23.8762675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.8763137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.8763701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.8764183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.8764772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.8765348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.8765916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.8766381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.8766828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.8767299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.8767945Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.8768623Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.8769119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.8769766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.8771040Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.8771842Z   warnings.warn(
2022-11-23T03:06:23.8773001Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.8773775Z   warnings.warn(
2022-11-23T03:06:23.8774096Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.8774501Z Traceback (most recent call last):
2022-11-23T03:06:23.8775047Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8775426Z     getattr(self, test_name)()
2022-11-23T03:06:23.8775945Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8776313Z     fn()
2022-11-23T03:06:23.8776971Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8777382Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8777792Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8778163Z     self._dist_train()
2022-11-23T03:06:23.8778527Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8779034Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8779453Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8779902Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8780423Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8780767Z     raise self.failureException(msg)
2022-11-23T03:06:23.8781063Z AssertionError: False is not true
2022-11-23T03:06:23.8781333Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.8781604Z dist init r=0, world=2
2022-11-23T03:06:23.8781935Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.8782439Z Traceback (most recent call last):
2022-11-23T03:06:23.8782977Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8783370Z     getattr(self, test_name)()
2022-11-23T03:06:23.8783877Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8784247Z     fn()
2022-11-23T03:06:23.8784728Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8785104Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8785651Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8786009Z     self._dist_train()
2022-11-23T03:06:23.8786377Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8786831Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8787233Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8787596Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8787932Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8788278Z     raise self.failureException(msg)
2022-11-23T03:06:23.8788583Z AssertionError: False is not true
2022-11-23T03:06:23.8788854Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.8789120Z dist init r=1, world=2
2022-11-23T03:06:23.8789450Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.8789925Z ERROR (4.113s)
2022-11-23T03:06:23.8790444Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 43
2022-11-23T03:06:23.8790874Z Traceback (most recent call last):
2022-11-23T03:06:23.8791400Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.8791775Z     self._join_processes(fn)
2022-11-23T03:06:23.8792304Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.8792946Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.8793484Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.8794072Z     raise RuntimeError(error)
2022-11-23T03:06:23.8794412Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.8794755Z Traceback (most recent call last):
2022-11-23T03:06:23.8795267Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8795657Z     getattr(self, test_name)()
2022-11-23T03:06:23.8796166Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8796514Z     fn()
2022-11-23T03:06:23.8797139Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8797510Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8797903Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8798314Z     self._dist_train()
2022-11-23T03:06:23.8798691Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8799185Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8799577Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8799953Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8800324Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8800657Z     raise self.failureException(msg)
2022-11-23T03:06:23.8800963Z AssertionError: False is not true
2022-11-23T03:06:23.8801142Z 
2022-11-23T03:06:23.8801297Z Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.8801606Z Traceback (most recent call last):
2022-11-23T03:06:23.8802108Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8802493Z     getattr(self, test_name)()
2022-11-23T03:06:23.8803162Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8803508Z     fn()
2022-11-23T03:06:23.8803987Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8804369Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8804771Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8805123Z     self._dist_train()
2022-11-23T03:06:23.8805498Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8806138Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8806528Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8806898Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8807255Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8807580Z     raise self.failureException(msg)
2022-11-23T03:06:23.8808052Z AssertionError: False is not true
2022-11-23T03:06:23.8808235Z 
2022-11-23T03:06:23.8808254Z 
2022-11-23T03:06:23.8808272Z 
2022-11-23T03:06:23.8808492Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 129782
2022-11-23T03:06:23.8808945Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 129783
2022-11-23T03:06:23.8809538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.8810044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.8810688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.8811173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.8811741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.8812188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.8812756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.8813205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.8813665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.8814158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.8814974Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.8815680Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.8816187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.8816647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.8818272Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.8819065Z   warnings.warn(
2022-11-23T03:06:23.8820202Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.8820980Z   warnings.warn(
2022-11-23T03:06:23.8821337Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.8821699Z Traceback (most recent call last):
2022-11-23T03:06:23.8822217Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8822621Z     getattr(self, test_name)()
2022-11-23T03:06:23.8823157Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8823513Z     fn()
2022-11-23T03:06:23.8824164Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8824553Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8824929Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8825299Z     self._dist_train()
2022-11-23T03:06:23.8825674Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8826161Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8826550Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8826923Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8827284Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8827859Z     raise self.failureException(msg)
2022-11-23T03:06:23.8828197Z AssertionError: False is not true
2022-11-23T03:06:23.8828509Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.8828863Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.8829226Z Traceback (most recent call last):
2022-11-23T03:06:23.8829773Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8830181Z     getattr(self, test_name)()
2022-11-23T03:06:23.8830685Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8831220Z     fn()
2022-11-23T03:06:23.8831694Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8832321Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8832738Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8833117Z     self._dist_train()
2022-11-23T03:06:23.8833586Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8834065Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8834494Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8834893Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8835244Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8835619Z     raise self.failureException(msg)
2022-11-23T03:06:23.8836088Z AssertionError: False is not true
2022-11-23T03:06:23.8836363Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.8836641Z dist init r=0, world=2
2022-11-23T03:06:23.8836896Z dist init r=1, world=2
2022-11-23T03:06:23.8837213Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.8837539Z ERROR (4.013s)
2022-11-23T03:06:23.8838067Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 42
2022-11-23T03:06:23.8838665Z Traceback (most recent call last):
2022-11-23T03:06:23.8839185Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.8839594Z     self._join_processes(fn)
2022-11-23T03:06:23.8840136Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.8840549Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.8841118Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.8841534Z     raise RuntimeError(error)
2022-11-23T03:06:23.8841897Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.8842230Z Traceback (most recent call last):
2022-11-23T03:06:23.8842769Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8843175Z     getattr(self, test_name)()
2022-11-23T03:06:23.8843677Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8844057Z     fn()
2022-11-23T03:06:23.8844544Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8845070Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8845470Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8845837Z     self._dist_train()
2022-11-23T03:06:23.8846208Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8846724Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8847153Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8847718Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8848070Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8848436Z     raise self.failureException(msg)
2022-11-23T03:06:23.8848751Z AssertionError: False is not true
2022-11-23T03:06:23.8848941Z 
2022-11-23T03:06:23.8848960Z 
2022-11-23T03:06:23.8848979Z 
2022-11-23T03:06:23.8849204Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 129933
2022-11-23T03:06:23.8849645Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 129934
2022-11-23T03:06:23.8850273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.8850737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.8851307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.8851859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.8852775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.8853237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.8853797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.8854272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.8854736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.8855242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.8855893Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.8857089Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.8857637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.8858096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.8859369Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.8860156Z   warnings.warn(
2022-11-23T03:06:23.8861372Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.8862212Z   warnings.warn(
2022-11-23T03:06:23.8862570Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.8862913Z Traceback (most recent call last):
2022-11-23T03:06:23.8863466Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8863872Z     getattr(self, test_name)()
2022-11-23T03:06:23.8864505Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8864894Z     fn()
2022-11-23T03:06:23.8865401Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8865788Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8866205Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8866584Z     self._dist_train()
2022-11-23T03:06:23.8866968Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8867447Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8867872Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8868274Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8868784Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8869149Z     raise self.failureException(msg)
2022-11-23T03:06:23.8869459Z AssertionError: False is not true
2022-11-23T03:06:23.8869810Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.8870274Z dist init r=1, world=2
2022-11-23T03:06:23.8870624Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.8870964Z Traceback (most recent call last):
2022-11-23T03:06:23.8871511Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8871917Z     getattr(self, test_name)()
2022-11-23T03:06:23.8872446Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8872958Z     fn()
2022-11-23T03:06:23.8873438Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8873999Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8874394Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8874778Z     self._dist_train()
2022-11-23T03:06:23.8875168Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8875667Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8876070Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8876464Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8876841Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8877186Z     raise self.failureException(msg)
2022-11-23T03:06:23.8877505Z AssertionError: False is not true
2022-11-23T03:06:23.8877811Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.8878073Z dist init r=0, world=2
2022-11-23T03:06:23.8878423Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.8878760Z ERROR (4.213s)
2022-11-23T03:06:23.8879284Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 41
2022-11-23T03:06:23.8879732Z Traceback (most recent call last):
2022-11-23T03:06:23.8880271Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.8880823Z     self._join_processes(fn)
2022-11-23T03:06:23.8881327Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.8881757Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.8882309Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.8882872Z     raise RuntimeError(error)
2022-11-23T03:06:23.8883224Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.8883627Z Traceback (most recent call last):
2022-11-23T03:06:23.8884189Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8884572Z     getattr(self, test_name)()
2022-11-23T03:06:23.8885098Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8885467Z     fn()
2022-11-23T03:06:23.8885933Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8886324Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8886734Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8887115Z     self._dist_train()
2022-11-23T03:06:23.8887479Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8888136Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8888548Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8888962Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8889330Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8889690Z     raise self.failureException(msg)
2022-11-23T03:06:23.8889979Z AssertionError: False is not true
2022-11-23T03:06:23.8890169Z 
2022-11-23T03:06:23.8890508Z Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.8890836Z Traceback (most recent call last):
2022-11-23T03:06:23.8891380Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8891760Z     getattr(self, test_name)()
2022-11-23T03:06:23.8892283Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8892657Z     fn()
2022-11-23T03:06:23.8893277Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8893665Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8894235Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8894592Z     self._dist_train()
2022-11-23T03:06:23.8894979Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8895478Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8895902Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8896270Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8896643Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8897347Z     raise self.failureException(msg)
2022-11-23T03:06:23.8897652Z AssertionError: False is not true
2022-11-23T03:06:23.8897844Z 
2022-11-23T03:06:23.8897863Z 
2022-11-23T03:06:23.8897885Z 
2022-11-23T03:06:23.8898116Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130084
2022-11-23T03:06:23.8898583Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 130085
2022-11-23T03:06:23.8899215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.8899803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.8900376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.8900839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.8901408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.8902083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.8902687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.8903168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.8903606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.8904115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.8904790Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.8905491Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.8906000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.8906624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.8907938Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.8908894Z   warnings.warn(
2022-11-23T03:06:23.8910113Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.8910869Z   warnings.warn(
2022-11-23T03:06:23.8911228Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.8911585Z Traceback (most recent call last):
2022-11-23T03:06:23.8912140Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8912521Z     getattr(self, test_name)()
2022-11-23T03:06:23.8913059Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8913437Z     fn()
2022-11-23T03:06:23.8913908Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8914308Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8914726Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8915088Z     self._dist_train()
2022-11-23T03:06:23.8915474Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8915981Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8916555Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8916916Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8917284Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8917643Z     raise self.failureException(msg)
2022-11-23T03:06:23.8917930Z AssertionError: False is not true
2022-11-23T03:06:23.8918232Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.8918596Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.8918930Z Traceback (most recent call last):
2022-11-23T03:06:23.8919508Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8919914Z     getattr(self, test_name)()
2022-11-23T03:06:23.8920607Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8920957Z     fn()
2022-11-23T03:06:23.8921446Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8921850Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8922237Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8922619Z     self._dist_train()
2022-11-23T03:06:23.8923009Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8923487Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8923916Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8924461Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8924900Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8925418Z     raise self.failureException(msg)
2022-11-23T03:06:23.8925739Z AssertionError: False is not true
2022-11-23T03:06:23.8926054Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.8926314Z dist init r=0, world=2
2022-11-23T03:06:23.8926571Z dist init r=1, world=2
2022-11-23T03:06:23.8926916Z Process 1 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.8927224Z ERROR (4.012s)
2022-11-23T03:06:23.8927777Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 40
2022-11-23T03:06:23.8928554Z Traceback (most recent call last):
2022-11-23T03:06:23.8929086Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.8929475Z     self._join_processes(fn)
2022-11-23T03:06:23.8930019Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.8930456Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.8931002Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.8931417Z     raise RuntimeError(error)
2022-11-23T03:06:23.8931764Z RuntimeError: Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.8932094Z Traceback (most recent call last):
2022-11-23T03:06:23.8932745Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8933145Z     getattr(self, test_name)()
2022-11-23T03:06:23.8933666Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8934022Z     fn()
2022-11-23T03:06:23.8934516Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8934914Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8935301Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8935681Z     self._dist_train()
2022-11-23T03:06:23.8936238Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8937158Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8937586Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8937977Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8938354Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8938698Z     raise self.failureException(msg)
2022-11-23T03:06:23.8939096Z AssertionError: False is not true
2022-11-23T03:06:23.8939295Z 
2022-11-23T03:06:23.8939317Z 
2022-11-23T03:06:23.8939336Z 
2022-11-23T03:06:23.8939569Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130235
2022-11-23T03:06:23.8940035Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 130236
2022-11-23T03:06:23.8940646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.8941110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.8941698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.8942158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.8942911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.8943362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.8943927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.8944624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.8945088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.8945595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.8946267Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.8946941Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.8947620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.8948089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.8949321Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.8950237Z   warnings.warn(
2022-11-23T03:06:23.8951404Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.8952185Z   warnings.warn(
2022-11-23T03:06:23.8952533Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.8952871Z Traceback (most recent call last):
2022-11-23T03:06:23.8953562Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8953951Z     getattr(self, test_name)()
2022-11-23T03:06:23.8954460Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8954802Z     fn()
2022-11-23T03:06:23.8955283Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8955668Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8956100Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8956483Z     self._dist_train()
2022-11-23T03:06:23.8956873Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8957365Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8957754Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8958306Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8958690Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8959034Z     raise self.failureException(msg)
2022-11-23T03:06:23.8959351Z AssertionError: False is not true
2022-11-23T03:06:23.8959658Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.8960029Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.8960390Z Traceback (most recent call last):
2022-11-23T03:06:23.8960939Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8961379Z     getattr(self, test_name)()
2022-11-23T03:06:23.8961926Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8962330Z     fn()
2022-11-23T03:06:23.8962829Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8963209Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8963627Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8964011Z     self._dist_train()
2022-11-23T03:06:23.8964373Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8964620Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8964859Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8965005Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8965207Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8965356Z     raise self.failureException(msg)
2022-11-23T03:06:23.8965651Z AssertionError: False is not true
2022-11-23T03:06:23.8965766Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.8965881Z dist init r=1, world=2
2022-11-23T03:06:23.8965994Z dist init r=0, world=2
2022-11-23T03:06:23.8966200Z Process 1 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.8966311Z ERROR (4.013s)
2022-11-23T03:06:23.8966702Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 39
2022-11-23T03:06:23.8966839Z Traceback (most recent call last):
2022-11-23T03:06:23.8967200Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.8967304Z     self._join_processes(fn)
2022-11-23T03:06:23.8967682Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.8967831Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.8968209Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.8968335Z     raise RuntimeError(error)
2022-11-23T03:06:23.8968522Z RuntimeError: Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.8968658Z Traceback (most recent call last):
2022-11-23T03:06:23.8969013Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8969114Z     getattr(self, test_name)()
2022-11-23T03:06:23.8969516Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8969629Z     fn()
2022-11-23T03:06:23.8969985Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8970115Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8970364Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8970481Z     self._dist_train()
2022-11-23T03:06:23.8970691Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8970929Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8971155Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8971294Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8971493Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8971640Z     raise self.failureException(msg)
2022-11-23T03:06:23.8971778Z AssertionError: False is not true
2022-11-23T03:06:23.8971843Z 
2022-11-23T03:06:23.8971860Z 
2022-11-23T03:06:23.8971878Z 
2022-11-23T03:06:23.8972107Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130386
2022-11-23T03:06:23.8972328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 130387
2022-11-23T03:06:23.8972845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.8973027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.8973418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.8973614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.8973984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.8974171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.8974553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.8974746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.8975090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.8975314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.8975718Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.8976120Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.8976354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.8976589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.8978218Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.8978344Z   warnings.warn(
2022-11-23T03:06:23.8979420Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.8979549Z   warnings.warn(
2022-11-23T03:06:23.8979760Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.8979903Z Traceback (most recent call last):
2022-11-23T03:06:23.8980254Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8980384Z     getattr(self, test_name)()
2022-11-23T03:06:23.8980755Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8981013Z     fn()
2022-11-23T03:06:23.8981364Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8981494Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8981749Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8981861Z     self._dist_train()
2022-11-23T03:06:23.8982187Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8982433Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8982664Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8982808Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8983183Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8983339Z     raise self.failureException(msg)
2022-11-23T03:06:23.8983484Z AssertionError: False is not true
2022-11-23T03:06:23.8983602Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.8983809Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.8983946Z Traceback (most recent call last):
2022-11-23T03:06:23.8984325Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8984457Z     getattr(self, test_name)()
2022-11-23T03:06:23.8984827Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8984929Z     fn()
2022-11-23T03:06:23.8985293Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8985401Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8985661Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8985779Z     self._dist_train()
2022-11-23T03:06:23.8986174Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8986410Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8986638Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8986782Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8986953Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8987098Z     raise self.failureException(msg)
2022-11-23T03:06:23.8987232Z AssertionError: False is not true
2022-11-23T03:06:23.8987367Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.8987481Z dist init r=1, world=2
2022-11-23T03:06:23.8987590Z dist init r=0, world=2
2022-11-23T03:06:23.8987967Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.8988055Z ERROR (4.213s)
2022-11-23T03:06:23.8988457Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 38
2022-11-23T03:06:23.8988595Z Traceback (most recent call last):
2022-11-23T03:06:23.8989011Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.8989149Z     self._join_processes(fn)
2022-11-23T03:06:23.8989544Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.8989699Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.8990090Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.8990228Z     raise RuntimeError(error)
2022-11-23T03:06:23.8990397Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.8990536Z Traceback (most recent call last):
2022-11-23T03:06:23.8991226Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8991354Z     getattr(self, test_name)()
2022-11-23T03:06:23.8991721Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8991823Z     fn()
2022-11-23T03:06:23.8992184Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8992341Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8992603Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8992722Z     self._dist_train()
2022-11-23T03:06:23.8992970Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8993219Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8993454Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8993600Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8993955Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8994082Z     raise self.failureException(msg)
2022-11-23T03:06:23.8994219Z AssertionError: False is not true
2022-11-23T03:06:23.8994241Z 
2022-11-23T03:06:23.8994571Z Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.8994715Z Traceback (most recent call last):
2022-11-23T03:06:23.8995085Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.8995213Z     getattr(self, test_name)()
2022-11-23T03:06:23.8995574Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.8995676Z     fn()
2022-11-23T03:06:23.8996014Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.8996142Z     return func(*args, **kwargs)
2022-11-23T03:06:23.8996396Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.8996518Z     self._dist_train()
2022-11-23T03:06:23.8996760Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.8997005Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.8997238Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.8997533Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.8997703Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.8997846Z     raise self.failureException(msg)
2022-11-23T03:06:23.8997984Z AssertionError: False is not true
2022-11-23T03:06:23.8998003Z 
2022-11-23T03:06:23.8998021Z 
2022-11-23T03:06:23.8998039Z 
2022-11-23T03:06:23.8998257Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130537
2022-11-23T03:06:23.8998475Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 130538
2022-11-23T03:06:23.8998892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.8999082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.8999459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.8999648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.8999982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9000220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9000585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9000773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9001017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9001259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9001707Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9002099Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9002305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9002526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9003505Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9003620Z   warnings.warn(
2022-11-23T03:06:23.9004578Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9004684Z   warnings.warn(
2022-11-23T03:06:23.9004880Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9005010Z Traceback (most recent call last):
2022-11-23T03:06:23.9005372Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9005496Z     getattr(self, test_name)()
2022-11-23T03:06:23.9005854Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9005931Z     fn()
2022-11-23T03:06:23.9006285Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9006410Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9006655Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9006768Z     self._dist_train()
2022-11-23T03:06:23.9007003Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9007239Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9007526Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9007650Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9007852Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9007999Z     raise self.failureException(msg)
2022-11-23T03:06:23.9008144Z AssertionError: False is not true
2022-11-23T03:06:23.9008312Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9008724Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9008868Z Traceback (most recent call last):
2022-11-23T03:06:23.9009224Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9009358Z     getattr(self, test_name)()
2022-11-23T03:06:23.9009724Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9009830Z     fn()
2022-11-23T03:06:23.9010242Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9010450Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9010710Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9010825Z     self._dist_train()
2022-11-23T03:06:23.9011045Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9011296Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9011531Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9011677Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9011878Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9012036Z     raise self.failureException(msg)
2022-11-23T03:06:23.9012179Z AssertionError: False is not true
2022-11-23T03:06:23.9012299Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9012420Z dist init r=0, world=2
2022-11-23T03:06:23.9012540Z dist init r=1, world=2
2022-11-23T03:06:23.9012748Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9012858Z ERROR (4.113s)
2022-11-23T03:06:23.9013258Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 37
2022-11-23T03:06:23.9013394Z Traceback (most recent call last):
2022-11-23T03:06:23.9013761Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9013866Z     self._join_processes(fn)
2022-11-23T03:06:23.9014252Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9014405Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9014794Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9014929Z     raise RuntimeError(error)
2022-11-23T03:06:23.9015116Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.9015249Z Traceback (most recent call last):
2022-11-23T03:06:23.9015763Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9015864Z     getattr(self, test_name)()
2022-11-23T03:06:23.9016208Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9016302Z     fn()
2022-11-23T03:06:23.9016645Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9017068Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9017392Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9017510Z     self._dist_train()
2022-11-23T03:06:23.9017723Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9017968Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9018190Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9018327Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9018517Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9018664Z     raise self.failureException(msg)
2022-11-23T03:06:23.9018797Z AssertionError: False is not true
2022-11-23T03:06:23.9018817Z 
2022-11-23T03:06:23.9018976Z Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.9019087Z Traceback (most recent call last):
2022-11-23T03:06:23.9019446Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9019565Z     getattr(self, test_name)()
2022-11-23T03:06:23.9019914Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9020077Z     fn()
2022-11-23T03:06:23.9020429Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9020721Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9020975Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9021067Z     self._dist_train()
2022-11-23T03:06:23.9021306Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9021546Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9021774Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9021915Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9022109Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9022258Z     raise self.failureException(msg)
2022-11-23T03:06:23.9022393Z AssertionError: False is not true
2022-11-23T03:06:23.9022413Z 
2022-11-23T03:06:23.9022432Z 
2022-11-23T03:06:23.9022450Z 
2022-11-23T03:06:23.9022652Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130688
2022-11-23T03:06:23.9022872Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 130689
2022-11-23T03:06:23.9023248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9023425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9023805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9024000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9024366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9024542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9025062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9025227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9025463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9025698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9026084Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9026514Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9026747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9026972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9027956Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9028069Z   warnings.warn(
2022-11-23T03:06:23.9029220Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9029381Z   warnings.warn(
2022-11-23T03:06:23.9029566Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9029704Z Traceback (most recent call last):
2022-11-23T03:06:23.9030074Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9030199Z     getattr(self, test_name)()
2022-11-23T03:06:23.9030562Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9030664Z     fn()
2022-11-23T03:06:23.9031026Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9031154Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9031389Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9031505Z     self._dist_train()
2022-11-23T03:06:23.9031743Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9032131Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9032523Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9032663Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9032856Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9033003Z     raise self.failureException(msg)
2022-11-23T03:06:23.9033119Z AssertionError: False is not true
2022-11-23T03:06:23.9033260Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9033463Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9033600Z Traceback (most recent call last):
2022-11-23T03:06:23.9033968Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9034089Z     getattr(self, test_name)()
2022-11-23T03:06:23.9034450Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9034531Z     fn()
2022-11-23T03:06:23.9034893Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9035018Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9035265Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9035375Z     self._dist_train()
2022-11-23T03:06:23.9035662Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9035911Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9036143Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9036264Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9036459Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9036605Z     raise self.failureException(msg)
2022-11-23T03:06:23.9036740Z AssertionError: False is not true
2022-11-23T03:06:23.9036875Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9036987Z dist init r=1, world=2
2022-11-23T03:06:23.9037094Z dist init r=0, world=2
2022-11-23T03:06:23.9037277Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9037382Z ERROR (4.113s)
2022-11-23T03:06:23.9037782Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 36
2022-11-23T03:06:23.9037917Z Traceback (most recent call last):
2022-11-23T03:06:23.9038335Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9038462Z     self._join_processes(fn)
2022-11-23T03:06:23.9038844Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9038992Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9039360Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9039489Z     raise RuntimeError(error)
2022-11-23T03:06:23.9039675Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.9039807Z Traceback (most recent call last):
2022-11-23T03:06:23.9040172Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9040296Z     getattr(self, test_name)()
2022-11-23T03:06:23.9040655Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9040751Z     fn()
2022-11-23T03:06:23.9041088Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9041212Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9041462Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9041573Z     self._dist_train()
2022-11-23T03:06:23.9041810Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9042047Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9042278Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9042416Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9042593Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9042897Z     raise self.failureException(msg)
2022-11-23T03:06:23.9043027Z AssertionError: False is not true
2022-11-23T03:06:23.9043047Z 
2022-11-23T03:06:23.9043201Z Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.9043330Z Traceback (most recent call last):
2022-11-23T03:06:23.9043683Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9043800Z     getattr(self, test_name)()
2022-11-23T03:06:23.9044132Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9044227Z     fn()
2022-11-23T03:06:23.9044573Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9044740Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9044987Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9045099Z     self._dist_train()
2022-11-23T03:06:23.9045329Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9045560Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9045761Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9045895Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9046083Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9046224Z     raise self.failureException(msg)
2022-11-23T03:06:23.9046358Z AssertionError: False is not true
2022-11-23T03:06:23.9046377Z 
2022-11-23T03:06:23.9046396Z 
2022-11-23T03:06:23.9046413Z 
2022-11-23T03:06:23.9046633Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130839
2022-11-23T03:06:23.9047079Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 130840
2022-11-23T03:06:23.9047457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9047635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9047996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9048191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9048556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9048727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9049102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9049291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9049539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9049783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9050334Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9050700Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9050922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9051143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9052125Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9052239Z   warnings.warn(
2022-11-23T03:06:23.9053376Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9053486Z   warnings.warn(
2022-11-23T03:06:23.9053738Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9053879Z Traceback (most recent call last):
2022-11-23T03:06:23.9054256Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9054361Z     getattr(self, test_name)()
2022-11-23T03:06:23.9054726Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9054824Z     fn()
2022-11-23T03:06:23.9055184Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9055311Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9055565Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9055678Z     self._dist_train()
2022-11-23T03:06:23.9055922Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9056140Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9056570Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9056885Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9057083Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9057225Z     raise self.failureException(msg)
2022-11-23T03:06:23.9057358Z AssertionError: False is not true
2022-11-23T03:06:23.9057494Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9057691Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9057802Z Traceback (most recent call last):
2022-11-23T03:06:23.9058339Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9058464Z     getattr(self, test_name)()
2022-11-23T03:06:23.9058832Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9058935Z     fn()
2022-11-23T03:06:23.9059294Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9059419Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9059650Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9059765Z     self._dist_train()
2022-11-23T03:06:23.9060003Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9060239Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9060469Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9060608Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9060806Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9060949Z     raise self.failureException(msg)
2022-11-23T03:06:23.9061069Z AssertionError: False is not true
2022-11-23T03:06:23.9061204Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9061316Z dist init r=0, world=2
2022-11-23T03:06:23.9061423Z dist init r=1, world=2
2022-11-23T03:06:23.9061625Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9061730Z ERROR (4.113s)
2022-11-23T03:06:23.9062182Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 35
2022-11-23T03:06:23.9062299Z Traceback (most recent call last):
2022-11-23T03:06:23.9062668Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9062791Z     self._join_processes(fn)
2022-11-23T03:06:23.9063243Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9063400Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9063792Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9063920Z     raise RuntimeError(error)
2022-11-23T03:06:23.9064111Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.9064226Z Traceback (most recent call last):
2022-11-23T03:06:23.9064589Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9064711Z     getattr(self, test_name)()
2022-11-23T03:06:23.9065071Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9065166Z     fn()
2022-11-23T03:06:23.9065524Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9065647Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9066124Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9066214Z     self._dist_train()
2022-11-23T03:06:23.9066444Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9066678Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9066903Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9067039Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9067235Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9067377Z     raise self.failureException(msg)
2022-11-23T03:06:23.9067510Z AssertionError: False is not true
2022-11-23T03:06:23.9067530Z 
2022-11-23T03:06:23.9067672Z Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.9067802Z Traceback (most recent call last):
2022-11-23T03:06:23.9068162Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9068285Z     getattr(self, test_name)()
2022-11-23T03:06:23.9068628Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9068722Z     fn()
2022-11-23T03:06:23.9069066Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9069168Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9069412Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9069521Z     self._dist_train()
2022-11-23T03:06:23.9069750Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9070159Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9070389Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9070531Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9070725Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9070850Z     raise self.failureException(msg)
2022-11-23T03:06:23.9070984Z AssertionError: False is not true
2022-11-23T03:06:23.9071003Z 
2022-11-23T03:06:23.9071022Z 
2022-11-23T03:06:23.9071041Z 
2022-11-23T03:06:23.9071261Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130990
2022-11-23T03:06:23.9071479Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 130991
2022-11-23T03:06:23.9071854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9072081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9072471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9072670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9073035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9073189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9073566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9073755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9074157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9074570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9074973Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9075439Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9075672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9075902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9076923Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9077042Z   warnings.warn(
2022-11-23T03:06:23.9078021Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9078137Z   warnings.warn(
2022-11-23T03:06:23.9078343Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9078478Z Traceback (most recent call last):
2022-11-23T03:06:23.9078847Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9078970Z     getattr(self, test_name)()
2022-11-23T03:06:23.9079337Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9079437Z     fn()
2022-11-23T03:06:23.9079796Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9079904Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9080154Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9080264Z     self._dist_train()
2022-11-23T03:06:23.9080502Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9080742Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9080971Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9081113Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9081489Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9081636Z     raise self.failureException(msg)
2022-11-23T03:06:23.9081774Z AssertionError: False is not true
2022-11-23T03:06:23.9081908Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9082018Z dist init r=1, world=2
2022-11-23T03:06:23.9082218Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9082353Z Traceback (most recent call last):
2022-11-23T03:06:23.9082712Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9082812Z     getattr(self, test_name)()
2022-11-23T03:06:23.9083161Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9083430Z     fn()
2022-11-23T03:06:23.9083792Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9083922Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9084172Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9084334Z     self._dist_train()
2022-11-23T03:06:23.9084555Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9084799Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9085029Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9085167Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9085359Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9085503Z     raise self.failureException(msg)
2022-11-23T03:06:23.9085637Z AssertionError: False is not true
2022-11-23T03:06:23.9085774Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9085866Z dist init r=0, world=2
2022-11-23T03:06:23.9086075Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9086183Z ERROR (4.113s)
2022-11-23T03:06:23.9086735Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 34
2022-11-23T03:06:23.9086864Z Traceback (most recent call last):
2022-11-23T03:06:23.9087212Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9087331Z     self._join_processes(fn)
2022-11-23T03:06:23.9087677Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9087819Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9088188Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9088311Z     raise RuntimeError(error)
2022-11-23T03:06:23.9088496Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.9088623Z Traceback (most recent call last):
2022-11-23T03:06:23.9088976Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9089094Z     getattr(self, test_name)()
2022-11-23T03:06:23.9089422Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9089516Z     fn()
2022-11-23T03:06:23.9089855Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9089975Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9090215Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9090323Z     self._dist_train()
2022-11-23T03:06:23.9090549Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9090826Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9091040Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9091181Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9091545Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9091693Z     raise self.failureException(msg)
2022-11-23T03:06:23.9091833Z AssertionError: False is not true
2022-11-23T03:06:23.9091854Z 
2022-11-23T03:06:23.9091874Z 
2022-11-23T03:06:23.9091892Z 
2022-11-23T03:06:23.9092111Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 369
2022-11-23T03:06:23.9092327Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 370
2022-11-23T03:06:23.9092703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9092882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9093244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9093510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9093883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9094060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9094580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9094763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9095182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9095430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9095835Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9096219Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9096449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9096677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9098043Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9098161Z   warnings.warn(
2022-11-23T03:06:23.9099127Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9099242Z   warnings.warn(
2022-11-23T03:06:23.9099436Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9099565Z Traceback (most recent call last):
2022-11-23T03:06:23.9099920Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9100020Z     getattr(self, test_name)()
2022-11-23T03:06:23.9100440Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9100549Z     fn()
2022-11-23T03:06:23.9100904Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9101032Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9101283Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9101395Z     self._dist_train()
2022-11-23T03:06:23.9101627Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9101838Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9102060Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9102196Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9102388Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9102528Z     raise self.failureException(msg)
2022-11-23T03:06:23.9102723Z AssertionError: False is not true
2022-11-23T03:06:23.9102858Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9102946Z dist init r=0, world=2
2022-11-23T03:06:23.9103146Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9103276Z Traceback (most recent call last):
2022-11-23T03:06:23.9103817Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9103940Z     getattr(self, test_name)()
2022-11-23T03:06:23.9104299Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9104401Z     fn()
2022-11-23T03:06:23.9104760Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9104871Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9105124Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9105236Z     self._dist_train()
2022-11-23T03:06:23.9105473Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9105711Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9105938Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9106076Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9106421Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9106545Z     raise self.failureException(msg)
2022-11-23T03:06:23.9106674Z AssertionError: False is not true
2022-11-23T03:06:23.9106807Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9106914Z dist init r=1, world=2
2022-11-23T03:06:23.9107117Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9107221Z ERROR (4.113s)
2022-11-23T03:06:23.9107611Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 33
2022-11-23T03:06:23.9107723Z Traceback (most recent call last):
2022-11-23T03:06:23.9108072Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9108193Z     self._join_processes(fn)
2022-11-23T03:06:23.9108559Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9108701Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9109256Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9109383Z     raise RuntimeError(error)
2022-11-23T03:06:23.9109619Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.9109740Z Traceback (most recent call last):
2022-11-23T03:06:23.9110165Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9110292Z     getattr(self, test_name)()
2022-11-23T03:06:23.9110654Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9110753Z     fn()
2022-11-23T03:06:23.9111113Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9111240Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9111493Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9111586Z     self._dist_train()
2022-11-23T03:06:23.9111828Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9112068Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9112351Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9112492Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9112688Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9112836Z     raise self.failureException(msg)
2022-11-23T03:06:23.9112952Z AssertionError: False is not true
2022-11-23T03:06:23.9112994Z 
2022-11-23T03:06:23.9113012Z 
2022-11-23T03:06:23.9113030Z 
2022-11-23T03:06:23.9113230Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 520
2022-11-23T03:06:23.9113450Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 521
2022-11-23T03:06:23.9113824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9114005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9114386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9114578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9114944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9115120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9115487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9115657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9116059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9116300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9116689Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9117077Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9117300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9117519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9118542Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9118662Z   warnings.warn(
2022-11-23T03:06:23.9119632Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9119743Z   warnings.warn(
2022-11-23T03:06:23.9119922Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9120058Z Traceback (most recent call last):
2022-11-23T03:06:23.9120412Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9120531Z     getattr(self, test_name)()
2022-11-23T03:06:23.9121053Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9121204Z     fn()
2022-11-23T03:06:23.9121568Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9121694Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9121926Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9122038Z     self._dist_train()
2022-11-23T03:06:23.9122279Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9122516Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9122744Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9122883Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9123083Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9123227Z     raise self.failureException(msg)
2022-11-23T03:06:23.9123348Z AssertionError: False is not true
2022-11-23T03:06:23.9123484Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9123595Z dist init r=1, world=2
2022-11-23T03:06:23.9123797Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9123931Z Traceback (most recent call last):
2022-11-23T03:06:23.9124297Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9124419Z     getattr(self, test_name)()
2022-11-23T03:06:23.9124756Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9124854Z     fn()
2022-11-23T03:06:23.9125362Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9125487Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9125727Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9125836Z     self._dist_train()
2022-11-23T03:06:23.9126066Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9126298Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9126502Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9126643Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9126832Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9126970Z     raise self.failureException(msg)
2022-11-23T03:06:23.9127100Z AssertionError: False is not true
2022-11-23T03:06:23.9127232Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9127338Z dist init r=0, world=2
2022-11-23T03:06:23.9127598Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9127711Z ERROR (4.213s)
2022-11-23T03:06:23.9128274Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 32
2022-11-23T03:06:23.9128411Z Traceback (most recent call last):
2022-11-23T03:06:23.9128776Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9128901Z     self._join_processes(fn)
2022-11-23T03:06:23.9129280Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9129426Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9129791Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9129918Z     raise RuntimeError(error)
2022-11-23T03:06:23.9130108Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.9130241Z Traceback (most recent call last):
2022-11-23T03:06:23.9130661Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9130786Z     getattr(self, test_name)()
2022-11-23T03:06:23.9131143Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9131240Z     fn()
2022-11-23T03:06:23.9131579Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9131704Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9131946Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9132056Z     self._dist_train()
2022-11-23T03:06:23.9132462Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9132862Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9133093Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9133213Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9133409Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9133553Z     raise self.failureException(msg)
2022-11-23T03:06:23.9133688Z AssertionError: False is not true
2022-11-23T03:06:23.9133708Z 
2022-11-23T03:06:23.9133870Z Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.9134005Z Traceback (most recent call last):
2022-11-23T03:06:23.9134370Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9134493Z     getattr(self, test_name)()
2022-11-23T03:06:23.9134842Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9134939Z     fn()
2022-11-23T03:06:23.9135302Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9135426Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9135675Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9135786Z     self._dist_train()
2022-11-23T03:06:23.9136023Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9136260Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9136623Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9136935Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9137132Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9137348Z     raise self.failureException(msg)
2022-11-23T03:06:23.9137485Z AssertionError: False is not true
2022-11-23T03:06:23.9137509Z 
2022-11-23T03:06:23.9137527Z 
2022-11-23T03:06:23.9137546Z 
2022-11-23T03:06:23.9137759Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 671
2022-11-23T03:06:23.9137971Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 672
2022-11-23T03:06:23.9138344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9138517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9138863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9139056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9139411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9139764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9140217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9140408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9140657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9140902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9141284Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9141683Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9141917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9142147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9143165Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9143280Z   warnings.warn(
2022-11-23T03:06:23.9144281Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9144394Z   warnings.warn(
2022-11-23T03:06:23.9144597Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9144882Z Traceback (most recent call last):
2022-11-23T03:06:23.9145241Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9145343Z     getattr(self, test_name)()
2022-11-23T03:06:23.9145694Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9145788Z     fn()
2022-11-23T03:06:23.9146136Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9146260Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9146552Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9146670Z     self._dist_train()
2022-11-23T03:06:23.9146902Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9147115Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9147337Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9147472Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9147661Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9147978Z     raise self.failureException(msg)
2022-11-23T03:06:23.9148114Z AssertionError: False is not true
2022-11-23T03:06:23.9148252Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9148435Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9148571Z Traceback (most recent call last):
2022-11-23T03:06:23.9148941Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9149114Z     getattr(self, test_name)()
2022-11-23T03:06:23.9149477Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9149573Z     fn()
2022-11-23T03:06:23.9149929Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9150058Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9150291Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9150399Z     self._dist_train()
2022-11-23T03:06:23.9150794Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9151020Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9151412Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9151553Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9151745Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9151888Z     raise self.failureException(msg)
2022-11-23T03:06:23.9152004Z AssertionError: False is not true
2022-11-23T03:06:23.9152139Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9152249Z dist init r=1, world=2
2022-11-23T03:06:23.9152357Z dist init r=0, world=2
2022-11-23T03:06:23.9152558Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9152661Z ERROR (4.113s)
2022-11-23T03:06:23.9153057Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 31
2022-11-23T03:06:23.9153173Z Traceback (most recent call last):
2022-11-23T03:06:23.9153537Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9153663Z     self._join_processes(fn)
2022-11-23T03:06:23.9154038Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9154184Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9154563Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9154687Z     raise RuntimeError(error)
2022-11-23T03:06:23.9154873Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.9154989Z Traceback (most recent call last):
2022-11-23T03:06:23.9155349Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9155471Z     getattr(self, test_name)()
2022-11-23T03:06:23.9155871Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9155973Z     fn()
2022-11-23T03:06:23.9156337Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9156462Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9156711Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9156803Z     self._dist_train()
2022-11-23T03:06:23.9157039Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9157275Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9157656Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9157791Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9157976Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9158117Z     raise self.failureException(msg)
2022-11-23T03:06:23.9158230Z AssertionError: False is not true
2022-11-23T03:06:23.9158310Z 
2022-11-23T03:06:23.9158452Z Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.9158790Z Traceback (most recent call last):
2022-11-23T03:06:23.9159153Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9159274Z     getattr(self, test_name)()
2022-11-23T03:06:23.9159627Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9159778Z     fn()
2022-11-23T03:06:23.9160134Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9160240Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9160491Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9160602Z     self._dist_train()
2022-11-23T03:06:23.9160838Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9161136Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9161367Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9161505Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9161698Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9161822Z     raise self.failureException(msg)
2022-11-23T03:06:23.9161958Z AssertionError: False is not true
2022-11-23T03:06:23.9161978Z 
2022-11-23T03:06:23.9162003Z 
2022-11-23T03:06:23.9162029Z 
2022-11-23T03:06:23.9162268Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 822
2022-11-23T03:06:23.9162485Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 823
2022-11-23T03:06:23.9162863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9163041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9163421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9163613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9163976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9164131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9164504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9164691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9164984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9165233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9165636Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9166031Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9166416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9166635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9167623Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9167760Z   warnings.warn(
2022-11-23T03:06:23.9168725Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9168832Z   warnings.warn(
2022-11-23T03:06:23.9169029Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9169161Z Traceback (most recent call last):
2022-11-23T03:06:23.9169517Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9169637Z     getattr(self, test_name)()
2022-11-23T03:06:23.9169987Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9170081Z     fn()
2022-11-23T03:06:23.9170589Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9170713Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9170959Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9171068Z     self._dist_train()
2022-11-23T03:06:23.9171305Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9171541Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9171772Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9171911Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9172088Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9172232Z     raise self.failureException(msg)
2022-11-23T03:06:23.9172364Z AssertionError: False is not true
2022-11-23T03:06:23.9172500Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9172698Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9172829Z Traceback (most recent call last):
2022-11-23T03:06:23.9173351Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9173451Z     getattr(self, test_name)()
2022-11-23T03:06:23.9173799Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9173890Z     fn()
2022-11-23T03:06:23.9174284Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9174413Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9174827Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9174938Z     self._dist_train()
2022-11-23T03:06:23.9175173Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9175396Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9175623Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9175761Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9175955Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9176101Z     raise self.failureException(msg)
2022-11-23T03:06:23.9176242Z AssertionError: False is not true
2022-11-23T03:06:23.9176378Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9176489Z dist init r=0, world=2
2022-11-23T03:06:23.9176627Z dist init r=1, world=2
2022-11-23T03:06:23.9177014Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9177122Z ERROR (4.113s)
2022-11-23T03:06:23.9177523Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 30
2022-11-23T03:06:23.9177809Z Traceback (most recent call last):
2022-11-23T03:06:23.9178158Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9178276Z     self._join_processes(fn)
2022-11-23T03:06:23.9178625Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9178940Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9179326Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9179455Z     raise RuntimeError(error)
2022-11-23T03:06:23.9179640Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.9179772Z Traceback (most recent call last):
2022-11-23T03:06:23.9180133Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9180253Z     getattr(self, test_name)()
2022-11-23T03:06:23.9180591Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9180685Z     fn()
2022-11-23T03:06:23.9181038Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9181159Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9181410Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9181519Z     self._dist_train()
2022-11-23T03:06:23.9181902Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9182130Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9182333Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9182466Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9182651Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9182789Z     raise self.failureException(msg)
2022-11-23T03:06:23.9182917Z AssertionError: False is not true
2022-11-23T03:06:23.9182936Z 
2022-11-23T03:06:23.9183093Z Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.9183220Z Traceback (most recent call last):
2022-11-23T03:06:23.9183794Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9183925Z     getattr(self, test_name)()
2022-11-23T03:06:23.9184296Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9184392Z     fn()
2022-11-23T03:06:23.9184747Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9184872Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9185119Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9185230Z     self._dist_train()
2022-11-23T03:06:23.9185450Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9185685Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9185915Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9186053Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9186312Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9186455Z     raise self.failureException(msg)
2022-11-23T03:06:23.9186589Z AssertionError: False is not true
2022-11-23T03:06:23.9186609Z 
2022-11-23T03:06:23.9186628Z 
2022-11-23T03:06:23.9186646Z 
2022-11-23T03:06:23.9186939Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 973
2022-11-23T03:06:23.9187376Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 974
2022-11-23T03:06:23.9188170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9188391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9188773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9188963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9189324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9189495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9189866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9190056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9190283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9190525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9191070Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9191454Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9191846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9192070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9193084Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9193197Z   warnings.warn(
2022-11-23T03:06:23.9194248Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9194362Z   warnings.warn(
2022-11-23T03:06:23.9194716Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9194847Z Traceback (most recent call last):
2022-11-23T03:06:23.9195355Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9195476Z     getattr(self, test_name)()
2022-11-23T03:06:23.9195831Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9195928Z     fn()
2022-11-23T03:06:23.9196291Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9196462Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9196709Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9196802Z     self._dist_train()
2022-11-23T03:06:23.9197039Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9197277Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9197502Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9197638Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9197828Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9197969Z     raise self.failureException(msg)
2022-11-23T03:06:23.9198254Z AssertionError: False is not true
2022-11-23T03:06:23.9198371Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9198558Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9198685Z Traceback (most recent call last):
2022-11-23T03:06:23.9199036Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9199150Z     getattr(self, test_name)()
2022-11-23T03:06:23.9199490Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9199579Z     fn()
2022-11-23T03:06:23.9199913Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9200017Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9200255Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9200358Z     self._dist_train()
2022-11-23T03:06:23.9200581Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9200813Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9201029Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9201160Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9201333Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9201469Z     raise self.failureException(msg)
2022-11-23T03:06:23.9201593Z AssertionError: False is not true
2022-11-23T03:06:23.9201720Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9201825Z dist init r=1, world=2
2022-11-23T03:06:23.9201926Z dist init r=0, world=2
2022-11-23T03:06:23.9202117Z Process 1 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9202202Z ERROR (4.213s)
2022-11-23T03:06:23.9202633Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 29
2022-11-23T03:06:23.9202765Z Traceback (most recent call last):
2022-11-23T03:06:23.9203116Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9203233Z     self._join_processes(fn)
2022-11-23T03:06:23.9203590Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9203730Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9204096Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9204201Z     raise RuntimeError(error)
2022-11-23T03:06:23.9204377Z RuntimeError: Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.9204505Z Traceback (most recent call last):
2022-11-23T03:06:23.9204857Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9204970Z     getattr(self, test_name)()
2022-11-23T03:06:23.9205373Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9205463Z     fn()
2022-11-23T03:06:23.9205798Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9205902Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9206136Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9206239Z     self._dist_train()
2022-11-23T03:06:23.9206460Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9206678Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9206893Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9207024Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9207206Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9207327Z     raise self.failureException(msg)
2022-11-23T03:06:23.9207452Z AssertionError: False is not true
2022-11-23T03:06:23.9207473Z 
2022-11-23T03:06:23.9207491Z 
2022-11-23T03:06:23.9207508Z 
2022-11-23T03:06:23.9207716Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1124
2022-11-23T03:06:23.9207917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1125
2022-11-23T03:06:23.9208274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9208432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9208791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9208966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9209495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9209653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9210016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9210250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9210493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9210734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9211131Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9211574Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9211810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9212037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9213049Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9213145Z   warnings.warn(
2022-11-23T03:06:23.9214152Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9214307Z   warnings.warn(
2022-11-23T03:06:23.9214501Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9214631Z Traceback (most recent call last):
2022-11-23T03:06:23.9214994Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9215111Z     getattr(self, test_name)()
2022-11-23T03:06:23.9215465Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9215555Z     fn()
2022-11-23T03:06:23.9215901Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9216018Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9216262Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9216520Z     self._dist_train()
2022-11-23T03:06:23.9216928Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9217172Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9217386Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9217517Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9217690Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9217827Z     raise self.failureException(msg)
2022-11-23T03:06:23.9217950Z AssertionError: False is not true
2022-11-23T03:06:23.9218080Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9218267Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9218395Z Traceback (most recent call last):
2022-11-23T03:06:23.9218741Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9218842Z     getattr(self, test_name)()
2022-11-23T03:06:23.9219184Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9219271Z     fn()
2022-11-23T03:06:23.9219610Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9219724Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9219961Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9220060Z     self._dist_train()
2022-11-23T03:06:23.9220356Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9220583Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9220802Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9220929Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9221110Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9221241Z     raise self.failureException(msg)
2022-11-23T03:06:23.9221541Z AssertionError: False is not true
2022-11-23T03:06:23.9221670Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9221762Z dist init r=1, world=2
2022-11-23T03:06:23.9221865Z dist init r=0, world=2
2022-11-23T03:06:23.9222062Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9222161Z ERROR (4.113s)
2022-11-23T03:06:23.9222554Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 28
2022-11-23T03:06:23.9222750Z Traceback (most recent call last):
2022-11-23T03:06:23.9223110Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9223229Z     self._join_processes(fn)
2022-11-23T03:06:23.9223586Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9223725Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9224099Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9224226Z     raise RuntimeError(error)
2022-11-23T03:06:23.9224404Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.9224532Z Traceback (most recent call last):
2022-11-23T03:06:23.9224890Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9225008Z     getattr(self, test_name)()
2022-11-23T03:06:23.9225349Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9225594Z     fn()
2022-11-23T03:06:23.9226123Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9226244Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9226486Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9226590Z     self._dist_train()
2022-11-23T03:06:23.9226826Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9227052Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9227266Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9227398Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9227586Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9227726Z     raise self.failureException(msg)
2022-11-23T03:06:23.9227854Z AssertionError: False is not true
2022-11-23T03:06:23.9227874Z 
2022-11-23T03:06:23.9228029Z Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.9228157Z Traceback (most recent call last):
2022-11-23T03:06:23.9228504Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9228780Z     getattr(self, test_name)()
2022-11-23T03:06:23.9229121Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9229210Z     fn()
2022-11-23T03:06:23.9229777Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9229903Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9230152Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9230257Z     self._dist_train()
2022-11-23T03:06:23.9230477Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9230712Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9230933Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9231066Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9231252Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9231391Z     raise self.failureException(msg)
2022-11-23T03:06:23.9231519Z AssertionError: False is not true
2022-11-23T03:06:23.9231538Z 
2022-11-23T03:06:23.9231556Z 
2022-11-23T03:06:23.9231579Z 
2022-11-23T03:06:23.9231789Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1275
2022-11-23T03:06:23.9232046Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1276
2022-11-23T03:06:23.9232405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9232571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9232940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9233129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9233485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9233653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9234020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9234205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9234433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9234669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9235063Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9235458Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9235681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9235905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9237067Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9237171Z   warnings.warn(
2022-11-23T03:06:23.9238130Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9238276Z   warnings.warn(
2022-11-23T03:06:23.9238473Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9238589Z Traceback (most recent call last):
2022-11-23T03:06:23.9238941Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9239054Z     getattr(self, test_name)()
2022-11-23T03:06:23.9239399Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9239488Z     fn()
2022-11-23T03:06:23.9239830Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9240111Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9240358Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9240452Z     self._dist_train()
2022-11-23T03:06:23.9240686Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9240969Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9241187Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9241320Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9241511Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9241648Z     raise self.failureException(msg)
2022-11-23T03:06:23.9241776Z AssertionError: False is not true
2022-11-23T03:06:23.9241894Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9242087Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9242214Z Traceback (most recent call last):
2022-11-23T03:06:23.9242574Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9242696Z     getattr(self, test_name)()
2022-11-23T03:06:23.9243055Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9243149Z     fn()
2022-11-23T03:06:23.9243642Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9243759Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9243998Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9244098Z     self._dist_train()
2022-11-23T03:06:23.9244323Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9244710Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9244930Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9245060Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9245239Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9245379Z     raise self.failureException(msg)
2022-11-23T03:06:23.9245507Z AssertionError: False is not true
2022-11-23T03:06:23.9245638Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9245742Z dist init r=1, world=2
2022-11-23T03:06:23.9245844Z dist init r=0, world=2
2022-11-23T03:06:23.9246042Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9246129Z ERROR (4.113s)
2022-11-23T03:06:23.9246520Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 27
2022-11-23T03:06:23.9246647Z Traceback (most recent call last):
2022-11-23T03:06:23.9247005Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9247120Z     self._join_processes(fn)
2022-11-23T03:06:23.9247686Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9247830Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9248199Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9248304Z     raise RuntimeError(error)
2022-11-23T03:06:23.9248477Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.9248596Z Traceback (most recent call last):
2022-11-23T03:06:23.9248941Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9249054Z     getattr(self, test_name)()
2022-11-23T03:06:23.9249390Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9249473Z     fn()
2022-11-23T03:06:23.9249815Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9249916Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9250199Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9250301Z     self._dist_train()
2022-11-23T03:06:23.9250524Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9250748Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9250960Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9251089Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9251271Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9251393Z     raise self.failureException(msg)
2022-11-23T03:06:23.9251518Z AssertionError: False is not true
2022-11-23T03:06:23.9251537Z 
2022-11-23T03:06:23.9251555Z 
2022-11-23T03:06:23.9251577Z 
2022-11-23T03:06:23.9251782Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1426
2022-11-23T03:06:23.9251987Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1427
2022-11-23T03:06:23.9252341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9252503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9252923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9253102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9253446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9253597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9253955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9254137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9254368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9254599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9254976Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9255347Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9255563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9255780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9256993Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9257121Z   warnings.warn(
2022-11-23T03:06:23.9258084Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9258191Z   warnings.warn(
2022-11-23T03:06:23.9258377Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9258559Z Traceback (most recent call last):
2022-11-23T03:06:23.9258915Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9259028Z     getattr(self, test_name)()
2022-11-23T03:06:23.9259554Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9259647Z     fn()
2022-11-23T03:06:23.9259986Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9260105Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9260346Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9260448Z     self._dist_train()
2022-11-23T03:06:23.9260683Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9260923Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9261144Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9261278Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9261454Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9261591Z     raise self.failureException(msg)
2022-11-23T03:06:23.9261720Z AssertionError: False is not true
2022-11-23T03:06:23.9261851Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9261956Z dist init r=1, world=2
2022-11-23T03:06:23.9262200Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9262327Z Traceback (most recent call last):
2022-11-23T03:06:23.9262682Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9262799Z     getattr(self, test_name)()
2022-11-23T03:06:23.9263149Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9263244Z     fn()
2022-11-23T03:06:23.9263596Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9263713Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9263952Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9264058Z     self._dist_train()
2022-11-23T03:06:23.9264279Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9264510Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9264728Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9264908Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9265101Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9265246Z     raise self.failureException(msg)
2022-11-23T03:06:23.9265377Z AssertionError: False is not true
2022-11-23T03:06:23.9265496Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9265603Z dist init r=0, world=2
2022-11-23T03:06:23.9265795Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9265896Z ERROR (4.213s)
2022-11-23T03:06:23.9266288Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 26
2022-11-23T03:06:23.9266415Z Traceback (most recent call last):
2022-11-23T03:06:23.9266917Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9267028Z     self._join_processes(fn)
2022-11-23T03:06:23.9267378Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9267624Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9267987Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9268108Z     raise RuntimeError(error)
2022-11-23T03:06:23.9268286Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.9268406Z Traceback (most recent call last):
2022-11-23T03:06:23.9268751Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9268864Z     getattr(self, test_name)()
2022-11-23T03:06:23.9269191Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9269276Z     fn()
2022-11-23T03:06:23.9269618Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9269730Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9269966Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9270065Z     self._dist_train()
2022-11-23T03:06:23.9270291Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9270515Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9270718Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9270845Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9271032Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9271173Z     raise self.failureException(msg)
2022-11-23T03:06:23.9271297Z AssertionError: False is not true
2022-11-23T03:06:23.9271317Z 
2022-11-23T03:06:23.9271469Z Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.9271597Z Traceback (most recent call last):
2022-11-23T03:06:23.9271938Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9272053Z     getattr(self, test_name)()
2022-11-23T03:06:23.9272396Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9272484Z     fn()
2022-11-23T03:06:23.9272824Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9272939Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9273353Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9273460Z     self._dist_train()
2022-11-23T03:06:23.9273738Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9273987Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9274214Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9274349Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9274540Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9274681Z     raise self.failureException(msg)
2022-11-23T03:06:23.9274811Z AssertionError: False is not true
2022-11-23T03:06:23.9274831Z 
2022-11-23T03:06:23.9274850Z 
2022-11-23T03:06:23.9274871Z 
2022-11-23T03:06:23.9275078Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1577
2022-11-23T03:06:23.9275295Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1578
2022-11-23T03:06:23.9275653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9275829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9276204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9276447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9276808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9276973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9277342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9277529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9277757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9278151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9278532Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9278915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9279313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9279529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9280548Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9280657Z   warnings.warn(
2022-11-23T03:06:23.9281654Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9281761Z   warnings.warn(
2022-11-23T03:06:23.9282105Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9282220Z Traceback (most recent call last):
2022-11-23T03:06:23.9282568Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9282681Z     getattr(self, test_name)()
2022-11-23T03:06:23.9283073Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9283170Z     fn()
2022-11-23T03:06:23.9283513Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9283631Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9283872Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9283961Z     self._dist_train()
2022-11-23T03:06:23.9284363Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9284591Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9284818Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9284952Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9285140Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9285277Z     raise self.failureException(msg)
2022-11-23T03:06:23.9285459Z AssertionError: False is not true
2022-11-23T03:06:23.9285578Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9285686Z dist init r=1, world=2
2022-11-23T03:06:23.9285883Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9286015Z Traceback (most recent call last):
2022-11-23T03:06:23.9286380Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9286500Z     getattr(self, test_name)()
2022-11-23T03:06:23.9286850Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9286928Z     fn()
2022-11-23T03:06:23.9287432Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9287554Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9287789Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9287896Z     self._dist_train()
2022-11-23T03:06:23.9288294Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9288532Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9288762Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9288883Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9289077Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9289218Z     raise self.failureException(msg)
2022-11-23T03:06:23.9289348Z AssertionError: False is not true
2022-11-23T03:06:23.9289484Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9289597Z dist init r=0, world=2
2022-11-23T03:06:23.9289803Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9289895Z ERROR (4.113s)
2022-11-23T03:06:23.9290296Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 25
2022-11-23T03:06:23.9290432Z Traceback (most recent call last):
2022-11-23T03:06:23.9290794Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9290915Z     self._join_processes(fn)
2022-11-23T03:06:23.9291431Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9291573Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9291942Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9292216Z     raise RuntimeError(error)
2022-11-23T03:06:23.9292453Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.9292596Z Traceback (most recent call last):
2022-11-23T03:06:23.9292959Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9293086Z     getattr(self, test_name)()
2022-11-23T03:06:23.9293442Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9293533Z     fn()
2022-11-23T03:06:23.9293885Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9293992Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9294237Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9294341Z     self._dist_train()
2022-11-23T03:06:23.9294582Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9294820Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9295243Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9295377Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9295726Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9295854Z     raise self.failureException(msg)
2022-11-23T03:06:23.9295986Z AssertionError: False is not true
2022-11-23T03:06:23.9296006Z 
2022-11-23T03:06:23.9296168Z Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.9296299Z Traceback (most recent call last):
2022-11-23T03:06:23.9296661Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9296971Z     getattr(self, test_name)()
2022-11-23T03:06:23.9297341Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9297419Z     fn()
2022-11-23T03:06:23.9297778Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9297898Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9298144Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9298252Z     self._dist_train()
2022-11-23T03:06:23.9298635Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9298860Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9299077Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9299194Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9299384Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9299517Z     raise self.failureException(msg)
2022-11-23T03:06:23.9299647Z AssertionError: False is not true
2022-11-23T03:06:23.9299666Z 
2022-11-23T03:06:23.9299684Z 
2022-11-23T03:06:23.9299702Z 
2022-11-23T03:06:23.9299909Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1728
2022-11-23T03:06:23.9300112Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1729
2022-11-23T03:06:23.9300467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9300634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9300998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9301166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9301588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9301762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9302126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9302306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9302541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9302774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9303154Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9303533Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9303741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9303960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9305010Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9305120Z   warnings.warn(
2022-11-23T03:06:23.9306084Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9306191Z   warnings.warn(
2022-11-23T03:06:23.9306382Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9306510Z Traceback (most recent call last):
2022-11-23T03:06:23.9306859Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9306973Z     getattr(self, test_name)()
2022-11-23T03:06:23.9307318Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9307394Z     fn()
2022-11-23T03:06:23.9307729Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9307847Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9308089Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9308195Z     self._dist_train()
2022-11-23T03:06:23.9308491Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9308725Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9308929Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9309063Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9309245Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9309376Z     raise self.failureException(msg)
2022-11-23T03:06:23.9309503Z AssertionError: False is not true
2022-11-23T03:06:23.9309802Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9309998Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9310219Z Traceback (most recent call last):
2022-11-23T03:06:23.9310581Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9310705Z     getattr(self, test_name)()
2022-11-23T03:06:23.9311064Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9311159Z     fn()
2022-11-23T03:06:23.9311517Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9311647Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9311895Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9312010Z     self._dist_train()
2022-11-23T03:06:23.9312229Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9312471Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9312699Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9312892Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9313089Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9313232Z     raise self.failureException(msg)
2022-11-23T03:06:23.9313363Z AssertionError: False is not true
2022-11-23T03:06:23.9313479Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9313585Z dist init r=0, world=2
2022-11-23T03:06:23.9313692Z dist init r=1, world=2
2022-11-23T03:06:23.9313890Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9313990Z ERROR (4.113s)
2022-11-23T03:06:23.9314385Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 24
2022-11-23T03:06:23.9314517Z Traceback (most recent call last):
2022-11-23T03:06:23.9314864Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9314986Z     self._join_processes(fn)
2022-11-23T03:06:23.9315358Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9315500Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9315879Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9316009Z     raise RuntimeError(error)
2022-11-23T03:06:23.9316194Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.9316325Z Traceback (most recent call last):
2022-11-23T03:06:23.9316819Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9316937Z     getattr(self, test_name)()
2022-11-23T03:06:23.9317289Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9317383Z     fn()
2022-11-23T03:06:23.9317725Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9317839Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9318076Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9318181Z     self._dist_train()
2022-11-23T03:06:23.9318390Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9318620Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9318839Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9318972Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9319209Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9319355Z     raise self.failureException(msg)
2022-11-23T03:06:23.9319492Z AssertionError: False is not true
2022-11-23T03:06:23.9319513Z 
2022-11-23T03:06:23.9319531Z 
2022-11-23T03:06:23.9319552Z 
2022-11-23T03:06:23.9319765Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1879
2022-11-23T03:06:23.9319971Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1880
2022-11-23T03:06:23.9320316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9320487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9320851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9321036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9321393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9321614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9322157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9322343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9322572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9322810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9323210Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9323601Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9323832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9324059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9325070Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9325178Z   warnings.warn(
2022-11-23T03:06:23.9326313Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9326417Z   warnings.warn(
2022-11-23T03:06:23.9326610Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9326724Z Traceback (most recent call last):
2022-11-23T03:06:23.9327075Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9327189Z     getattr(self, test_name)()
2022-11-23T03:06:23.9327534Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9327624Z     fn()
2022-11-23T03:06:23.9327967Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9328083Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9328364Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9328464Z     self._dist_train()
2022-11-23T03:06:23.9328688Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9328919Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9329136Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9329266Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9329449Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9329587Z     raise self.failureException(msg)
2022-11-23T03:06:23.9329715Z AssertionError: False is not true
2022-11-23T03:06:23.9330008Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9330204Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9330339Z Traceback (most recent call last):
2022-11-23T03:06:23.9330699Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9330880Z     getattr(self, test_name)()
2022-11-23T03:06:23.9331242Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9331336Z     fn()
2022-11-23T03:06:23.9331677Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9331799Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9332045Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9332150Z     self._dist_train()
2022-11-23T03:06:23.9332383Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9332621Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9332843Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9332979Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9333474Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9333611Z     raise self.failureException(msg)
2022-11-23T03:06:23.9333742Z AssertionError: False is not true
2022-11-23T03:06:23.9333872Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9333980Z dist init r=1, world=2
2022-11-23T03:06:23.9334087Z dist init r=0, world=2
2022-11-23T03:06:23.9334284Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9334373Z ERROR (4.113s)
2022-11-23T03:06:23.9334767Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 23
2022-11-23T03:06:23.9334896Z Traceback (most recent call last):
2022-11-23T03:06:23.9335256Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9335377Z     self._join_processes(fn)
2022-11-23T03:06:23.9335752Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9335895Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9336273Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9336382Z     raise RuntimeError(error)
2022-11-23T03:06:23.9336561Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.9336868Z Traceback (most recent call last):
2022-11-23T03:06:23.9337244Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9337364Z     getattr(self, test_name)()
2022-11-23T03:06:23.9337788Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9337889Z     fn()
2022-11-23T03:06:23.9338243Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9338349Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9338593Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9338699Z     self._dist_train()
2022-11-23T03:06:23.9338932Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9339167Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9339393Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9339528Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9339720Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9339849Z     raise self.failureException(msg)
2022-11-23T03:06:23.9340045Z AssertionError: False is not true
2022-11-23T03:06:23.9340065Z 
2022-11-23T03:06:23.9340084Z 
2022-11-23T03:06:23.9340103Z 
2022-11-23T03:06:23.9340319Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2030
2022-11-23T03:06:23.9340535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2031
2022-11-23T03:06:23.9340902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9341070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9341445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9341631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9341991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9342151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9342518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9342703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9342945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9343185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9343733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9344115Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9344338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9344544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9345524Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9345633Z   warnings.warn(
2022-11-23T03:06:23.9346644Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9346761Z   warnings.warn(
2022-11-23T03:06:23.9346957Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9347089Z Traceback (most recent call last):
2022-11-23T03:06:23.9347446Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9347563Z     getattr(self, test_name)()
2022-11-23T03:06:23.9347912Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9348010Z     fn()
2022-11-23T03:06:23.9348340Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9348462Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9348704Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9348861Z     self._dist_train()
2022-11-23T03:06:23.9349090Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9349316Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9349535Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9349670Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9349841Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9349981Z     raise self.failureException(msg)
2022-11-23T03:06:23.9350114Z AssertionError: False is not true
2022-11-23T03:06:23.9350250Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9350443Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9350576Z Traceback (most recent call last):
2022-11-23T03:06:23.9350930Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9351036Z     getattr(self, test_name)()
2022-11-23T03:06:23.9351381Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9351472Z     fn()
2022-11-23T03:06:23.9351817Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9351936Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9352176Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9352278Z     self._dist_train()
2022-11-23T03:06:23.9352508Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9352722Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9352943Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9353081Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9353264Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9353403Z     raise self.failureException(msg)
2022-11-23T03:06:23.9353531Z AssertionError: False is not true
2022-11-23T03:06:23.9353664Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9353754Z dist init r=1, world=2
2022-11-23T03:06:23.9353865Z dist init r=0, world=2
2022-11-23T03:06:23.9354063Z Process 1 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9354171Z ERROR (4.013s)
2022-11-23T03:06:23.9354557Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 22
2022-11-23T03:06:23.9354691Z Traceback (most recent call last):
2022-11-23T03:06:23.9355087Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9355220Z     self._join_processes(fn)
2022-11-23T03:06:23.9355570Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9355715Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9356086Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9356214Z     raise RuntimeError(error)
2022-11-23T03:06:23.9356399Z RuntimeError: Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.9356536Z Traceback (most recent call last):
2022-11-23T03:06:23.9356891Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9357019Z     getattr(self, test_name)()
2022-11-23T03:06:23.9357352Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9357502Z     fn()
2022-11-23T03:06:23.9357853Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9357977Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9358221Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9358336Z     self._dist_train()
2022-11-23T03:06:23.9358568Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9358778Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9359003Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9359142Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9359343Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9359485Z     raise self.failureException(msg)
2022-11-23T03:06:23.9359798Z AssertionError: False is not true
2022-11-23T03:06:23.9359819Z 
2022-11-23T03:06:23.9359838Z 
2022-11-23T03:06:23.9359856Z 
2022-11-23T03:06:23.9360081Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2181
2022-11-23T03:06:23.9360305Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2182
2022-11-23T03:06:23.9360682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9360838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9361221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9361420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9361791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9361970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9362390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9362585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9362835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9363086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9363471Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9363871Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9364152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9364393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9365421Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9365539Z   warnings.warn(
2022-11-23T03:06:23.9366541Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9366700Z   warnings.warn(
2022-11-23T03:06:23.9367062Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9367199Z Traceback (most recent call last):
2022-11-23T03:06:23.9367555Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9367656Z     getattr(self, test_name)()
2022-11-23T03:06:23.9368006Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9368108Z     fn()
2022-11-23T03:06:23.9368451Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9368579Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9368827Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9368941Z     self._dist_train()
2022-11-23T03:06:23.9369150Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9369568Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9369794Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9369938Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9370134Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9370288Z     raise self.failureException(msg)
2022-11-23T03:06:23.9370429Z AssertionError: False is not true
2022-11-23T03:06:23.9370573Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9370755Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9370900Z Traceback (most recent call last):
2022-11-23T03:06:23.9371273Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9371403Z     getattr(self, test_name)()
2022-11-23T03:06:23.9371845Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9371946Z     fn()
2022-11-23T03:06:23.9372306Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9372434Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9372665Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9372779Z     self._dist_train()
2022-11-23T03:06:23.9373020Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9373310Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9373549Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9373700Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9373898Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9374024Z     raise self.failureException(msg)
2022-11-23T03:06:23.9374164Z AssertionError: False is not true
2022-11-23T03:06:23.9374304Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9374423Z dist init r=0, world=2
2022-11-23T03:06:23.9374692Z dist init r=1, world=2
2022-11-23T03:06:23.9374890Z Process 1 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9375168Z ERROR (4.112s)
2022-11-23T03:06:23.9375548Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 21
2022-11-23T03:06:23.9375689Z Traceback (most recent call last):
2022-11-23T03:06:23.9376059Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9376236Z     self._join_processes(fn)
2022-11-23T03:06:23.9376619Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9377084Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9377491Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9377621Z     raise RuntimeError(error)
2022-11-23T03:06:23.9377789Z RuntimeError: Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.9377930Z Traceback (most recent call last):
2022-11-23T03:06:23.9378293Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9378418Z     getattr(self, test_name)()
2022-11-23T03:06:23.9378782Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9378887Z     fn()
2022-11-23T03:06:23.9379248Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9379373Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9379605Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9379722Z     self._dist_train()
2022-11-23T03:06:23.9379959Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9380198Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9380434Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9380582Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9380783Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9380928Z     raise self.failureException(msg)
2022-11-23T03:06:23.9381047Z AssertionError: False is not true
2022-11-23T03:06:23.9381068Z 
2022-11-23T03:06:23.9381086Z 
2022-11-23T03:06:23.9381126Z 
2022-11-23T03:06:23.9381325Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2332
2022-11-23T03:06:23.9381544Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2333
2022-11-23T03:06:23.9381920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9382100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9382630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9382818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9383246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9383434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9383779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9383967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9384213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9384623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9385026Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9385424Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9385660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9385961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9386987Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9387106Z   warnings.warn(
2022-11-23T03:06:23.9388243Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9388362Z   warnings.warn(
2022-11-23T03:06:23.9388540Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9401087Z Traceback (most recent call last):
2022-11-23T03:06:23.9401661Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9401781Z     getattr(self, test_name)()
2022-11-23T03:06:23.9402137Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9402225Z     fn()
2022-11-23T03:06:23.9402569Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9402695Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9402924Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9403032Z     self._dist_train()
2022-11-23T03:06:23.9403260Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9403495Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9403886Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9404022Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9404213Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9404341Z     raise self.failureException(msg)
2022-11-23T03:06:23.9404474Z AssertionError: False is not true
2022-11-23T03:06:23.9404608Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9404720Z dist init r=0, world=2
2022-11-23T03:06:23.9405023Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9405163Z Traceback (most recent call last):
2022-11-23T03:06:23.9405539Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9405658Z     getattr(self, test_name)()
2022-11-23T03:06:23.9405999Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9406095Z     fn()
2022-11-23T03:06:23.9406451Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9406567Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9406959Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9407065Z     self._dist_train()
2022-11-23T03:06:23.9407294Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9407523Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9407802Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9407936Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9408118Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9408251Z     raise self.failureException(msg)
2022-11-23T03:06:23.9408374Z AssertionError: False is not true
2022-11-23T03:06:23.9408501Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9408608Z dist init r=1, world=2
2022-11-23T03:06:23.9408785Z Process 1 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9408884Z ERROR (4.013s)
2022-11-23T03:06:23.9409269Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 20
2022-11-23T03:06:23.9409394Z Traceback (most recent call last):
2022-11-23T03:06:23.9409742Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9409863Z     self._join_processes(fn)
2022-11-23T03:06:23.9410484Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9410627Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9410997Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9411118Z     raise RuntimeError(error)
2022-11-23T03:06:23.9411297Z RuntimeError: Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.9411426Z Traceback (most recent call last):
2022-11-23T03:06:23.9411783Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9411903Z     getattr(self, test_name)()
2022-11-23T03:06:23.9412263Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9412345Z     fn()
2022-11-23T03:06:23.9412699Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9412818Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9413066Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9413172Z     self._dist_train()
2022-11-23T03:06:23.9413406Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9413637Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9413858Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9413979Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9414230Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9414382Z     raise self.failureException(msg)
2022-11-23T03:06:23.9414513Z AssertionError: False is not true
2022-11-23T03:06:23.9414535Z 
2022-11-23T03:06:23.9414555Z 
2022-11-23T03:06:23.9414574Z 
2022-11-23T03:06:23.9414787Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2483
2022-11-23T03:06:23.9415002Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2484
2022-11-23T03:06:23.9415372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9415538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9415914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9416087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9416450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9417085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9417478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9417658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9417895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9418127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9418509Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9418898Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9419104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9419323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9420305Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9420410Z   warnings.warn(
2022-11-23T03:06:23.9421369Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9421478Z   warnings.warn(
2022-11-23T03:06:23.9421669Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9421794Z Traceback (most recent call last):
2022-11-23T03:06:23.9422322Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9422442Z     getattr(self, test_name)()
2022-11-23T03:06:23.9422786Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9422879Z     fn()
2022-11-23T03:06:23.9423232Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9423478Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9423736Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9423848Z     self._dist_train()
2022-11-23T03:06:23.9424082Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9424317Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9424528Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9424663Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9424854Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9424994Z     raise self.failureException(msg)
2022-11-23T03:06:23.9425127Z AssertionError: False is not true
2022-11-23T03:06:23.9425258Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9425365Z dist init r=1, world=2
2022-11-23T03:06:23.9425553Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9425752Z Traceback (most recent call last):
2022-11-23T03:06:23.9426275Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9426389Z     getattr(self, test_name)()
2022-11-23T03:06:23.9426734Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9426823Z     fn()
2022-11-23T03:06:23.9427167Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9427281Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9427508Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9427607Z     self._dist_train()
2022-11-23T03:06:23.9427835Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9428061Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9428443Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9428580Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9428769Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9428909Z     raise self.failureException(msg)
2022-11-23T03:06:23.9429026Z AssertionError: False is not true
2022-11-23T03:06:23.9429158Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9429266Z dist init r=0, world=2
2022-11-23T03:06:23.9429463Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9429564Z ERROR (4.113s)
2022-11-23T03:06:23.9429955Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 19
2022-11-23T03:06:23.9430087Z Traceback (most recent call last):
2022-11-23T03:06:23.9430446Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9430560Z     self._join_processes(fn)
2022-11-23T03:06:23.9430932Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9431071Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9431449Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9431565Z     raise RuntimeError(error)
2022-11-23T03:06:23.9431747Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.9431876Z Traceback (most recent call last):
2022-11-23T03:06:23.9432224Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9432393Z     getattr(self, test_name)()
2022-11-23T03:06:23.9432759Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9432854Z     fn()
2022-11-23T03:06:23.9433210Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9433332Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9433904Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9434011Z     self._dist_train()
2022-11-23T03:06:23.9434229Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9434463Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9434687Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9434819Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9435013Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9435210Z     raise self.failureException(msg)
2022-11-23T03:06:23.9435341Z AssertionError: False is not true
2022-11-23T03:06:23.9435363Z 
2022-11-23T03:06:23.9435522Z Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.9435637Z Traceback (most recent call last):
2022-11-23T03:06:23.9435998Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9436116Z     getattr(self, test_name)()
2022-11-23T03:06:23.9436470Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9436562Z     fn()
2022-11-23T03:06:23.9436916Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9437033Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9437269Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9437379Z     self._dist_train()
2022-11-23T03:06:23.9437761Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9437986Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9438199Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9438328Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9438520Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9438658Z     raise self.failureException(msg)
2022-11-23T03:06:23.9438770Z AssertionError: False is not true
2022-11-23T03:06:23.9438809Z 
2022-11-23T03:06:23.9438827Z 
2022-11-23T03:06:23.9438845Z 
2022-11-23T03:06:23.9439042Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2634
2022-11-23T03:06:23.9439253Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2635
2022-11-23T03:06:23.9439617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9439787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9440153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9440337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9440866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9441042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9441412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9441631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9441877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9442128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9442528Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9442923Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9443156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9443384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9444403Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9444567Z   warnings.warn(
2022-11-23T03:06:23.9445707Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9445818Z   warnings.warn(
2022-11-23T03:06:23.9445996Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9446129Z Traceback (most recent call last):
2022-11-23T03:06:23.9446485Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9446606Z     getattr(self, test_name)()
2022-11-23T03:06:23.9446957Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9447050Z     fn()
2022-11-23T03:06:23.9447396Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9447514Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9447739Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9447847Z     self._dist_train()
2022-11-23T03:06:23.9448072Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9448485Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9448710Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9448854Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9449048Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9449174Z     raise self.failureException(msg)
2022-11-23T03:06:23.9449306Z AssertionError: False is not true
2022-11-23T03:06:23.9449442Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9449550Z dist init r=0, world=2
2022-11-23T03:06:23.9449754Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9449890Z Traceback (most recent call last):
2022-11-23T03:06:23.9450254Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9450378Z     getattr(self, test_name)()
2022-11-23T03:06:23.9450765Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9450871Z     fn()
2022-11-23T03:06:23.9451386Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9451511Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9451751Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9451861Z     self._dist_train()
2022-11-23T03:06:23.9452090Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9452302Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9452524Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9452658Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9452848Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9452986Z     raise self.failureException(msg)
2022-11-23T03:06:23.9453170Z AssertionError: False is not true
2022-11-23T03:06:23.9453302Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9453409Z dist init r=1, world=2
2022-11-23T03:06:23.9453587Z Process 1 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9453691Z ERROR (4.113s)
2022-11-23T03:06:23.9454076Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 18
2022-11-23T03:06:23.9454206Z Traceback (most recent call last):
2022-11-23T03:06:23.9454551Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9454673Z     self._join_processes(fn)
2022-11-23T03:06:23.9455038Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9455167Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9455540Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9455670Z     raise RuntimeError(error)
2022-11-23T03:06:23.9455844Z RuntimeError: Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.9455968Z Traceback (most recent call last):
2022-11-23T03:06:23.9456321Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9456437Z     getattr(self, test_name)()
2022-11-23T03:06:23.9456984Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9457065Z     fn()
2022-11-23T03:06:23.9457420Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9457546Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9457789Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9457896Z     self._dist_train()
2022-11-23T03:06:23.9458126Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9458354Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9458579Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9458694Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9459050Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9459197Z     raise self.failureException(msg)
2022-11-23T03:06:23.9459333Z AssertionError: False is not true
2022-11-23T03:06:23.9459354Z 
2022-11-23T03:06:23.9459373Z 
2022-11-23T03:06:23.9459391Z 
2022-11-23T03:06:23.9459683Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2785
2022-11-23T03:06:23.9459910Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2786
2022-11-23T03:06:23.9460289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9460463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9460839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9461014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9461447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9461620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9461998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9462234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9462585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9462830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9463240Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9463639Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9463853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9464082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9465098Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9465214Z   warnings.warn(
2022-11-23T03:06:23.9466211Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9466317Z   warnings.warn(
2022-11-23T03:06:23.9466522Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9466655Z Traceback (most recent call last):
2022-11-23T03:06:23.9467025Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9467311Z     getattr(self, test_name)()
2022-11-23T03:06:23.9467648Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9467745Z     fn()
2022-11-23T03:06:23.9468093Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9468213Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9468454Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9468565Z     self._dist_train()
2022-11-23T03:06:23.9468793Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9469073Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9469283Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9469422Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9469611Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9469755Z     raise self.failureException(msg)
2022-11-23T03:06:23.9469892Z AssertionError: False is not true
2022-11-23T03:06:23.9470026Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9470215Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9470343Z Traceback (most recent call last):
2022-11-23T03:06:23.9470858Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9470983Z     getattr(self, test_name)()
2022-11-23T03:06:23.9471346Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9471439Z     fn()
2022-11-23T03:06:23.9471863Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9471983Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9472232Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9472326Z     self._dist_train()
2022-11-23T03:06:23.9472564Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9472801Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9473027Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9473161Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9473355Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9473502Z     raise self.failureException(msg)
2022-11-23T03:06:23.9473792Z AssertionError: False is not true
2022-11-23T03:06:23.9473909Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9474015Z dist init r=1, world=2
2022-11-23T03:06:23.9474120Z dist init r=0, world=2
2022-11-23T03:06:23.9474313Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9474420Z ERROR (4.113s)
2022-11-23T03:06:23.9474808Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 17
2022-11-23T03:06:23.9474936Z Traceback (most recent call last):
2022-11-23T03:06:23.9475268Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9475388Z     self._join_processes(fn)
2022-11-23T03:06:23.9475926Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9476076Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9476459Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9476591Z     raise RuntimeError(error)
2022-11-23T03:06:23.9476777Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.9476916Z Traceback (most recent call last):
2022-11-23T03:06:23.9477260Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9477381Z     getattr(self, test_name)()
2022-11-23T03:06:23.9477739Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9477832Z     fn()
2022-11-23T03:06:23.9478186Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9478362Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9478772Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9478887Z     self._dist_train()
2022-11-23T03:06:23.9479097Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9479330Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9479549Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9479680Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9480044Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9480187Z     raise self.failureException(msg)
2022-11-23T03:06:23.9480324Z AssertionError: False is not true
2022-11-23T03:06:23.9480344Z 
2022-11-23T03:06:23.9480504Z Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.9480624Z Traceback (most recent call last):
2022-11-23T03:06:23.9480990Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9481161Z     getattr(self, test_name)()
2022-11-23T03:06:23.9481525Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9481623Z     fn()
2022-11-23T03:06:23.9481983Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9482105Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9482334Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9482446Z     self._dist_train()
2022-11-23T03:06:23.9482832Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9483064Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9483284Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9483420Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9483610Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9483750Z     raise self.failureException(msg)
2022-11-23T03:06:23.9483862Z AssertionError: False is not true
2022-11-23T03:06:23.9483881Z 
2022-11-23T03:06:23.9483922Z 
2022-11-23T03:06:23.9483940Z 
2022-11-23T03:06:23.9484132Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2936
2022-11-23T03:06:23.9484336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2937
2022-11-23T03:06:23.9484700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9485040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9485423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9485615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9485977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9486148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9486501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9486689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9486931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9487178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9487624Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9488190Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9488415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9488634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9489618Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9489725Z   warnings.warn(
2022-11-23T03:06:23.9490684Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9490836Z   warnings.warn(
2022-11-23T03:06:23.9491014Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9491140Z Traceback (most recent call last):
2022-11-23T03:06:23.9491492Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9491610Z     getattr(self, test_name)()
2022-11-23T03:06:23.9491955Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9492054Z     fn()
2022-11-23T03:06:23.9492391Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9492498Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9492906Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9493018Z     self._dist_train()
2022-11-23T03:06:23.9493254Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9493490Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9493716Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9493856Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9494054Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9494183Z     raise self.failureException(msg)
2022-11-23T03:06:23.9494317Z AssertionError: False is not true
2022-11-23T03:06:23.9494456Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9494655Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9494789Z Traceback (most recent call last):
2022-11-23T03:06:23.9495149Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9495271Z     getattr(self, test_name)()
2022-11-23T03:06:23.9495783Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9495862Z     fn()
2022-11-23T03:06:23.9496210Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9496507Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9497155Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9497284Z     self._dist_train()
2022-11-23T03:06:23.9497527Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9497774Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9498003Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9498124Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9498317Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9498466Z     raise self.failureException(msg)
2022-11-23T03:06:23.9498599Z AssertionError: False is not true
2022-11-23T03:06:23.9498734Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9498844Z dist init r=0, world=2
2022-11-23T03:06:23.9498950Z dist init r=1, world=2
2022-11-23T03:06:23.9499135Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9499241Z ERROR (4.113s)
2022-11-23T03:06:23.9499636Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 16
2022-11-23T03:06:23.9499835Z Traceback (most recent call last):
2022-11-23T03:06:23.9500353Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9500476Z     self._join_processes(fn)
2022-11-23T03:06:23.9500840Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9500963Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9501335Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9501453Z     raise RuntimeError(error)
2022-11-23T03:06:23.9501633Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.9501764Z Traceback (most recent call last):
2022-11-23T03:06:23.9502116Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9502236Z     getattr(self, test_name)()
2022-11-23T03:06:23.9502581Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9502657Z     fn()
2022-11-23T03:06:23.9502997Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9503117Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9503363Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9503470Z     self._dist_train()
2022-11-23T03:06:23.9503697Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9503933Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9504156Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9504274Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9504458Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9504595Z     raise self.failureException(msg)
2022-11-23T03:06:23.9504716Z AssertionError: False is not true
2022-11-23T03:06:23.9504735Z 
2022-11-23T03:06:23.9504890Z Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.9505011Z Traceback (most recent call last):
2022-11-23T03:06:23.9505365Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9505484Z     getattr(self, test_name)()
2022-11-23T03:06:23.9505813Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9505902Z     fn()
2022-11-23T03:06:23.9506294Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9506424Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9506668Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9506778Z     self._dist_train()
2022-11-23T03:06:23.9507001Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9507229Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9507430Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9507555Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9507739Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9507878Z     raise self.failureException(msg)
2022-11-23T03:06:23.9508008Z AssertionError: False is not true
2022-11-23T03:06:23.9508027Z 
2022-11-23T03:06:23.9508045Z 
2022-11-23T03:06:23.9508107Z 
2022-11-23T03:06:23.9508327Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3087
2022-11-23T03:06:23.9508537Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3088
2022-11-23T03:06:23.9508893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9509045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9509410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9509595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9510124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9510350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9510729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9510923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9511174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9511420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9511802Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9512195Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9512417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9512642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9513659Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9513771Z   warnings.warn(
2022-11-23T03:06:23.9514813Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9514927Z   warnings.warn(
2022-11-23T03:06:23.9515136Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9515265Z Traceback (most recent call last):
2022-11-23T03:06:23.9515622Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9515728Z     getattr(self, test_name)()
2022-11-23T03:06:23.9516079Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9516165Z     fn()
2022-11-23T03:06:23.9516515Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9516627Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9516864Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9516966Z     self._dist_train()
2022-11-23T03:06:23.9517354Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9517621Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9517844Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9517976Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9518159Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9518288Z     raise self.failureException(msg)
2022-11-23T03:06:23.9518417Z AssertionError: False is not true
2022-11-23T03:06:23.9518547Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9518637Z dist init r=1, world=2
2022-11-23T03:06:23.9518829Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9518956Z Traceback (most recent call last):
2022-11-23T03:06:23.9519311Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9519429Z     getattr(self, test_name)()
2022-11-23T03:06:23.9519773Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9519863Z     fn()
2022-11-23T03:06:23.9520206Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9520308Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9520545Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9520644Z     self._dist_train()
2022-11-23T03:06:23.9520871Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9521095Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9521315Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9521452Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9521622Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9521757Z     raise self.failureException(msg)
2022-11-23T03:06:23.9521883Z AssertionError: False is not true
2022-11-23T03:06:23.9522015Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9522115Z dist init r=0, world=2
2022-11-23T03:06:23.9522311Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9522579Z ERROR (4.013s)
2022-11-23T03:06:23.9522980Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 15
2022-11-23T03:06:23.9523096Z Traceback (most recent call last):
2022-11-23T03:06:23.9523450Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9523626Z     self._join_processes(fn)
2022-11-23T03:06:23.9524011Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9524155Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9524538Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9524663Z     raise RuntimeError(error)
2022-11-23T03:06:23.9524846Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.9524961Z Traceback (most recent call last):
2022-11-23T03:06:23.9525321Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9525443Z     getattr(self, test_name)()
2022-11-23T03:06:23.9525798Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9525896Z     fn()
2022-11-23T03:06:23.9526247Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9526416Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9526649Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9526756Z     self._dist_train()
2022-11-23T03:06:23.9526987Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9527224Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9527450Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9527590Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9527780Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9527918Z     raise self.failureException(msg)
2022-11-23T03:06:23.9528038Z AssertionError: False is not true
2022-11-23T03:06:23.9528057Z 
2022-11-23T03:06:23.9528095Z 
2022-11-23T03:06:23.9528116Z 
2022-11-23T03:06:23.9528317Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3238
2022-11-23T03:06:23.9528530Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3239
2022-11-23T03:06:23.9529065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9529230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9529592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9529777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9530123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9530294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9530814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9531007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9531256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9531490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9531889Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9532282Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9532508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9532783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9533813Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9533931Z   warnings.warn(
2022-11-23T03:06:23.9534931Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9535098Z   warnings.warn(
2022-11-23T03:06:23.9535286Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9535418Z Traceback (most recent call last):
2022-11-23T03:06:23.9535783Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9535906Z     getattr(self, test_name)()
2022-11-23T03:06:23.9536265Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9536361Z     fn()
2022-11-23T03:06:23.9536922Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9537040Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9537293Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9537402Z     self._dist_train()
2022-11-23T03:06:23.9537636Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9538036Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9538256Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9538390Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9538579Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9538700Z     raise self.failureException(msg)
2022-11-23T03:06:23.9538829Z AssertionError: False is not true
2022-11-23T03:06:23.9538958Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9539062Z dist init r=1, world=2
2022-11-23T03:06:23.9539251Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9539383Z Traceback (most recent call last):
2022-11-23T03:06:23.9539735Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9539840Z     getattr(self, test_name)()
2022-11-23T03:06:23.9540186Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9540279Z     fn()
2022-11-23T03:06:23.9540625Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9540740Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9540980Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9541251Z     self._dist_train()
2022-11-23T03:06:23.9541486Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9541707Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9542007Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9542158Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9542348Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9542488Z     raise self.failureException(msg)
2022-11-23T03:06:23.9542620Z AssertionError: False is not true
2022-11-23T03:06:23.9542755Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9542863Z dist init r=0, world=2
2022-11-23T03:06:23.9543050Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9543154Z ERROR (4.013s)
2022-11-23T03:06:23.9543550Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 14
2022-11-23T03:06:23.9543679Z Traceback (most recent call last):
2022-11-23T03:06:23.9544037Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9544323Z     self._join_processes(fn)
2022-11-23T03:06:23.9544686Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9545044Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9545434Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9545558Z     raise RuntimeError(error)
2022-11-23T03:06:23.9545744Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.9545879Z Traceback (most recent call last):
2022-11-23T03:06:23.9546239Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9546360Z     getattr(self, test_name)()
2022-11-23T03:06:23.9546718Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9546800Z     fn()
2022-11-23T03:06:23.9547157Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9547276Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9547521Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9547624Z     self._dist_train()
2022-11-23T03:06:23.9548010Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9548239Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9548454Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9548569Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9548754Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9548893Z     raise self.failureException(msg)
2022-11-23T03:06:23.9549024Z AssertionError: False is not true
2022-11-23T03:06:23.9549043Z 
2022-11-23T03:06:23.9549064Z 
2022-11-23T03:06:23.9549082Z 
2022-11-23T03:06:23.9549292Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3389
2022-11-23T03:06:23.9549498Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3390
2022-11-23T03:06:23.9549857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9550020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9550376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9550544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9550894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9551122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9551490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9551674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9551910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9552145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9552526Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9552907Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9553112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9553331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9554370Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9554481Z   warnings.warn(
2022-11-23T03:06:23.9555448Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9555556Z   warnings.warn(
2022-11-23T03:06:23.9555752Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9555876Z Traceback (most recent call last):
2022-11-23T03:06:23.9556229Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9556345Z     getattr(self, test_name)()
2022-11-23T03:06:23.9556676Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9556765Z     fn()
2022-11-23T03:06:23.9557106Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9557222Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9557468Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9557576Z     self._dist_train()
2022-11-23T03:06:23.9557809Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9558033Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9558237Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9558370Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9558553Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9558692Z     raise self.failureException(msg)
2022-11-23T03:06:23.9558818Z AssertionError: False is not true
2022-11-23T03:06:23.9558945Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9559138Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9559250Z Traceback (most recent call last):
2022-11-23T03:06:23.9559642Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9559770Z     getattr(self, test_name)()
2022-11-23T03:06:23.9560114Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9560212Z     fn()
2022-11-23T03:06:23.9560732Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9560856Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9561106Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9561199Z     self._dist_train()
2022-11-23T03:06:23.9561433Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9561671Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9561900Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9562038Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9562372Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9562517Z     raise self.failureException(msg)
2022-11-23T03:06:23.9562646Z AssertionError: False is not true
2022-11-23T03:06:23.9562764Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9562870Z dist init r=0, world=2
2022-11-23T03:06:23.9562979Z dist init r=1, world=2
2022-11-23T03:06:23.9563182Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9563290Z ERROR (4.213s)
2022-11-23T03:06:23.9563687Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 13
2022-11-23T03:06:23.9563817Z Traceback (most recent call last):
2022-11-23T03:06:23.9564162Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9564287Z     self._join_processes(fn)
2022-11-23T03:06:23.9564658Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9564802Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9565182Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9565306Z     raise RuntimeError(error)
2022-11-23T03:06:23.9565487Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.9565618Z Traceback (most recent call last):
2022-11-23T03:06:23.9565960Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9566078Z     getattr(self, test_name)()
2022-11-23T03:06:23.9566432Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9566531Z     fn()
2022-11-23T03:06:23.9566878Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9566999Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9567243Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9567351Z     self._dist_train()
2022-11-23T03:06:23.9567724Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9567948Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9568161Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9568291Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9568477Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9568664Z     raise self.failureException(msg)
2022-11-23T03:06:23.9568800Z AssertionError: False is not true
2022-11-23T03:06:23.9568820Z 
2022-11-23T03:06:23.9568963Z Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.9569087Z Traceback (most recent call last):
2022-11-23T03:06:23.9569439Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9569555Z     getattr(self, test_name)()
2022-11-23T03:06:23.9569895Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9569987Z     fn()
2022-11-23T03:06:23.9570325Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9570443Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9570668Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9570778Z     self._dist_train()
2022-11-23T03:06:23.9571003Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9571277Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9571490Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9571620Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9571804Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9571935Z     raise self.failureException(msg)
2022-11-23T03:06:23.9572045Z AssertionError: False is not true
2022-11-23T03:06:23.9572064Z 
2022-11-23T03:06:23.9572092Z 
2022-11-23T03:06:23.9572110Z 
2022-11-23T03:06:23.9572302Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3540
2022-11-23T03:06:23.9572506Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3541
2022-11-23T03:06:23.9572866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9573029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9573387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9573743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9574101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9574267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9574620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9574805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9575049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9575286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9575681Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9576072Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9576298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9576518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9577812Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9577938Z   warnings.warn(
2022-11-23T03:06:23.9579094Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9579200Z   warnings.warn(
2022-11-23T03:06:23.9579381Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9579506Z Traceback (most recent call last):
2022-11-23T03:06:23.9579859Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9579972Z     getattr(self, test_name)()
2022-11-23T03:06:23.9580562Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9580652Z     fn()
2022-11-23T03:06:23.9581001Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9581107Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9581354Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9581458Z     self._dist_train()
2022-11-23T03:06:23.9581687Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9581918Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9582142Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9582272Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9582466Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9582592Z     raise self.failureException(msg)
2022-11-23T03:06:23.9582719Z AssertionError: False is not true
2022-11-23T03:06:23.9582846Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9583187Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9583308Z Traceback (most recent call last):
2022-11-23T03:06:23.9583655Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9583765Z     getattr(self, test_name)()
2022-11-23T03:06:23.9584107Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9584183Z     fn()
2022-11-23T03:06:23.9584527Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9584644Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9584881Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9584980Z     self._dist_train()
2022-11-23T03:06:23.9585371Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9585603Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9585815Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9585952Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9586138Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9586277Z     raise self.failureException(msg)
2022-11-23T03:06:23.9586403Z AssertionError: False is not true
2022-11-23T03:06:23.9586582Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9586698Z dist init r=1, world=2
2022-11-23T03:06:23.9586790Z dist init r=0, world=2
2022-11-23T03:06:23.9586991Z Process 1 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9587090Z ERROR (4.113s)
2022-11-23T03:06:23.9587480Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 12
2022-11-23T03:06:23.9587611Z Traceback (most recent call last):
2022-11-23T03:06:23.9587963Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9588080Z     self._join_processes(fn)
2022-11-23T03:06:23.9588786Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9588915Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9589296Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9589417Z     raise RuntimeError(error)
2022-11-23T03:06:23.9589652Z RuntimeError: Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.9589782Z Traceback (most recent call last):
2022-11-23T03:06:23.9590137Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9590255Z     getattr(self, test_name)()
2022-11-23T03:06:23.9590610Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9590688Z     fn()
2022-11-23T03:06:23.9591039Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9591155Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9591397Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9591659Z     self._dist_train()
2022-11-23T03:06:23.9591881Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9592107Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9592321Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9592437Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9592621Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9592752Z     raise self.failureException(msg)
2022-11-23T03:06:23.9592871Z AssertionError: False is not true
2022-11-23T03:06:23.9592890Z 
2022-11-23T03:06:23.9592909Z 
2022-11-23T03:06:23.9592926Z 
2022-11-23T03:06:23.9593301Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3691
2022-11-23T03:06:23.9593509Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3692
2022-11-23T03:06:23.9593877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9594050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9594423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9594597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9594954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9595121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9595484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9595667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9595954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9596355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9596907Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9597287Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9597509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9597730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9598746Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9598917Z   warnings.warn(
2022-11-23T03:06:23.9600054Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9600157Z   warnings.warn(
2022-11-23T03:06:23.9600346Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9600469Z Traceback (most recent call last):
2022-11-23T03:06:23.9600823Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9600934Z     getattr(self, test_name)()
2022-11-23T03:06:23.9601269Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9601359Z     fn()
2022-11-23T03:06:23.9601699Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9601816Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9602056Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9602160Z     self._dist_train()
2022-11-23T03:06:23.9602387Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9602615Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9602822Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9602954Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9603140Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9603278Z     raise self.failureException(msg)
2022-11-23T03:06:23.9603404Z AssertionError: False is not true
2022-11-23T03:06:23.9603530Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9603717Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9603828Z Traceback (most recent call last):
2022-11-23T03:06:23.9604177Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9604289Z     getattr(self, test_name)()
2022-11-23T03:06:23.9604627Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9604711Z     fn()
2022-11-23T03:06:23.9605100Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9605225Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9605462Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9605550Z     self._dist_train()
2022-11-23T03:06:23.9605775Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9606002Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9606214Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9606340Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9606520Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9606654Z     raise self.failureException(msg)
2022-11-23T03:06:23.9606776Z AssertionError: False is not true
2022-11-23T03:06:23.9606894Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9606999Z dist init r=1, world=2
2022-11-23T03:06:23.9607146Z dist init r=0, world=2
2022-11-23T03:06:23.9607339Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9607435Z ERROR (4.113s)
2022-11-23T03:06:23.9607814Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 11
2022-11-23T03:06:23.9607936Z Traceback (most recent call last):
2022-11-23T03:06:23.9608269Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9608477Z     self._join_processes(fn)
2022-11-23T03:06:23.9608891Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9609025Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9609393Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9609511Z     raise RuntimeError(error)
2022-11-23T03:06:23.9609686Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.9609808Z Traceback (most recent call last):
2022-11-23T03:06:23.9610142Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9610295Z     getattr(self, test_name)()
2022-11-23T03:06:23.9610643Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9610910Z     fn()
2022-11-23T03:06:23.9611262Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9611378Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9611626Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9611734Z     self._dist_train()
2022-11-23T03:06:23.9611952Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9612191Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9612412Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9612543Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9612735Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9612876Z     raise self.failureException(msg)
2022-11-23T03:06:23.9613005Z AssertionError: False is not true
2022-11-23T03:06:23.9613026Z 
2022-11-23T03:06:23.9613170Z Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.9613295Z Traceback (most recent call last):
2022-11-23T03:06:23.9613656Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9613827Z     getattr(self, test_name)()
2022-11-23T03:06:23.9614188Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9614283Z     fn()
2022-11-23T03:06:23.9614631Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9614747Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9614979Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9615079Z     self._dist_train()
2022-11-23T03:06:23.9615307Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9615534Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9615755Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9615892Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9616079Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9616268Z     raise self.failureException(msg)
2022-11-23T03:06:23.9616383Z AssertionError: False is not true
2022-11-23T03:06:23.9616403Z 
2022-11-23T03:06:23.9616422Z 
2022-11-23T03:06:23.9616452Z 
2022-11-23T03:06:23.9616653Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3842
2022-11-23T03:06:23.9617063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3843
2022-11-23T03:06:23.9617607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9617774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9618136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9618320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9618668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9618837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9619180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9619364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9619594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9619823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9620207Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9620590Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9620809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9621024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9622000Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9622101Z   warnings.warn(
2022-11-23T03:06:23.9623311Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9623427Z   warnings.warn(
2022-11-23T03:06:23.9623612Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9623747Z Traceback (most recent call last):
2022-11-23T03:06:23.9624114Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9624234Z     getattr(self, test_name)()
2022-11-23T03:06:23.9624588Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9624685Z     fn()
2022-11-23T03:06:23.9625045Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9625152Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9625459Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9625566Z     self._dist_train()
2022-11-23T03:06:23.9625799Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9626036Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9626269Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9626401Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9626591Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9626716Z     raise self.failureException(msg)
2022-11-23T03:06:23.9626999Z AssertionError: False is not true
2022-11-23T03:06:23.9627126Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9627232Z dist init r=0, world=2
2022-11-23T03:06:23.9627428Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9627557Z Traceback (most recent call last):
2022-11-23T03:06:23.9627903Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9628004Z     getattr(self, test_name)()
2022-11-23T03:06:23.9628348Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9628438Z     fn()
2022-11-23T03:06:23.9628787Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9628898Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9629134Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9629240Z     self._dist_train()
2022-11-23T03:06:23.9629468Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9629683Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9629897Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9630027Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9630206Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9630339Z     raise self.failureException(msg)
2022-11-23T03:06:23.9630464Z AssertionError: False is not true
2022-11-23T03:06:23.9630587Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9630676Z dist init r=1, world=2
2022-11-23T03:06:23.9630865Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9631140Z ERROR (4.113s)
2022-11-23T03:06:23.9631587Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 10
2022-11-23T03:06:23.9631725Z Traceback (most recent call last):
2022-11-23T03:06:23.9632086Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9632216Z     self._join_processes(fn)
2022-11-23T03:06:23.9632587Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9632715Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9633098Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9633218Z     raise RuntimeError(error)
2022-11-23T03:06:23.9633395Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.9633522Z Traceback (most recent call last):
2022-11-23T03:06:23.9633884Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9634000Z     getattr(self, test_name)()
2022-11-23T03:06:23.9634411Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9634489Z     fn()
2022-11-23T03:06:23.9634847Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9634966Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9635215Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9635322Z     self._dist_train()
2022-11-23T03:06:23.9635560Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9635790Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9636018Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9636142Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9636331Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9636472Z     raise self.failureException(msg)
2022-11-23T03:06:23.9636601Z AssertionError: False is not true
2022-11-23T03:06:23.9636621Z 
2022-11-23T03:06:23.9636640Z 
2022-11-23T03:06:23.9636658Z 
2022-11-23T03:06:23.9636869Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3993
2022-11-23T03:06:23.9637082Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3994
2022-11-23T03:06:23.9637451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9637623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9637994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9638170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9638528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9638703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9639077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9639267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9639505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9639744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9640139Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9640578Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9640822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9641050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9642071Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9642181Z   warnings.warn(
2022-11-23T03:06:23.9643175Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9643328Z   warnings.warn(
2022-11-23T03:06:23.9643526Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9643662Z Traceback (most recent call last):
2022-11-23T03:06:23.9644029Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9644147Z     getattr(self, test_name)()
2022-11-23T03:06:23.9644491Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9644584Z     fn()
2022-11-23T03:06:23.9644945Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9645067Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9645316Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9645422Z     self._dist_train()
2022-11-23T03:06:23.9645654Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9646036Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9646242Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9646373Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9646555Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9646685Z     raise self.failureException(msg)
2022-11-23T03:06:23.9646807Z AssertionError: False is not true
2022-11-23T03:06:23.9646939Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9647043Z dist init r=0, world=2
2022-11-23T03:06:23.9647221Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9647350Z Traceback (most recent call last):
2022-11-23T03:06:23.9647700Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9647817Z     getattr(self, test_name)()
2022-11-23T03:06:23.9648159Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9648249Z     fn()
2022-11-23T03:06:23.9648598Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9648715Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9648938Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9649036Z     self._dist_train()
2022-11-23T03:06:23.9649306Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9649551Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9649769Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9649897Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9650075Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9650212Z     raise self.failureException(msg)
2022-11-23T03:06:23.9650326Z AssertionError: False is not true
2022-11-23T03:06:23.9650450Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9650552Z dist init r=1, world=2
2022-11-23T03:06:23.9650743Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9650839Z ERROR (4.113s)
2022-11-23T03:06:23.9651220Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 9
2022-11-23T03:06:23.9651342Z Traceback (most recent call last):
2022-11-23T03:06:23.9651726Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9651848Z     self._join_processes(fn)
2022-11-23T03:06:23.9652212Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9652346Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9652889Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9653010Z     raise RuntimeError(error)
2022-11-23T03:06:23.9653197Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.9653327Z Traceback (most recent call last):
2022-11-23T03:06:23.9653675Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9653793Z     getattr(self, test_name)()
2022-11-23T03:06:23.9654149Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9654240Z     fn()
2022-11-23T03:06:23.9654595Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9654718Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9654967Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9655058Z     self._dist_train()
2022-11-23T03:06:23.9655285Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9655515Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9655739Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9655875Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9656063Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9656206Z     raise self.failureException(msg)
2022-11-23T03:06:23.9656334Z AssertionError: False is not true
2022-11-23T03:06:23.9656354Z 
2022-11-23T03:06:23.9656499Z Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.9656632Z Traceback (most recent call last):
2022-11-23T03:06:23.9657420Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9657543Z     getattr(self, test_name)()
2022-11-23T03:06:23.9658053Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9658143Z     fn()
2022-11-23T03:06:23.9658481Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9658674Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9658905Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9659012Z     self._dist_train()
2022-11-23T03:06:23.9659240Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9659465Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9659687Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9659817Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9660001Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9660132Z     raise self.failureException(msg)
2022-11-23T03:06:23.9660243Z AssertionError: False is not true
2022-11-23T03:06:23.9660262Z 
2022-11-23T03:06:23.9660281Z 
2022-11-23T03:06:23.9660311Z 
2022-11-23T03:06:23.9660508Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4144
2022-11-23T03:06:23.9660966Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4145
2022-11-23T03:06:23.9661341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9661512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9661887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9662072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9662487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9662657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9663022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9663214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9663455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9663692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9664088Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9664475Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9664699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9664918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9665930Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9666040Z   warnings.warn(
2022-11-23T03:06:23.9667047Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9667140Z   warnings.warn(
2022-11-23T03:06:23.9667390Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9667529Z Traceback (most recent call last):
2022-11-23T03:06:23.9668054Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9668171Z     getattr(self, test_name)()
2022-11-23T03:06:23.9668519Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9668616Z     fn()
2022-11-23T03:06:23.9668955Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9669057Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9669295Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9669394Z     self._dist_train()
2022-11-23T03:06:23.9669621Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9669847Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9670110Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9670243Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9670427Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9670719Z     raise self.failureException(msg)
2022-11-23T03:06:23.9670847Z AssertionError: False is not true
2022-11-23T03:06:23.9670981Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9671090Z dist init r=1, world=2
2022-11-23T03:06:23.9671284Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9671414Z Traceback (most recent call last):
2022-11-23T03:06:23.9671776Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9671884Z     getattr(self, test_name)()
2022-11-23T03:06:23.9672237Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9672335Z     fn()
2022-11-23T03:06:23.9672690Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9672810Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9673052Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9673154Z     self._dist_train()
2022-11-23T03:06:23.9673386Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9673604Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9673824Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9673957Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9674146Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9674288Z     raise self.failureException(msg)
2022-11-23T03:06:23.9674413Z AssertionError: False is not true
2022-11-23T03:06:23.9674541Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9674634Z dist init r=0, world=2
2022-11-23T03:06:23.9674978Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9675070Z ERROR (4.213s)
2022-11-23T03:06:23.9675628Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 8
2022-11-23T03:06:23.9675756Z Traceback (most recent call last):
2022-11-23T03:06:23.9676103Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9676218Z     self._join_processes(fn)
2022-11-23T03:06:23.9676634Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9676768Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9677151Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9677270Z     raise RuntimeError(error)
2022-11-23T03:06:23.9677454Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.9677570Z Traceback (most recent call last):
2022-11-23T03:06:23.9677924Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9678038Z     getattr(self, test_name)()
2022-11-23T03:06:23.9678391Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9678468Z     fn()
2022-11-23T03:06:23.9678820Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9678937Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9679180Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9679332Z     self._dist_train()
2022-11-23T03:06:23.9679562Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9679799Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9680022Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9680297Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9680653Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9680788Z     raise self.failureException(msg)
2022-11-23T03:06:23.9680914Z AssertionError: False is not true
2022-11-23T03:06:23.9680934Z 
2022-11-23T03:06:23.9681088Z Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.9681216Z Traceback (most recent call last):
2022-11-23T03:06:23.9681575Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9681687Z     getattr(self, test_name)()
2022-11-23T03:06:23.9682038Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9682125Z     fn()
2022-11-23T03:06:23.9682472Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9682589Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9682829Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9682928Z     self._dist_train()
2022-11-23T03:06:23.9683156Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9683530Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9683744Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9683874Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9684050Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9684179Z     raise self.failureException(msg)
2022-11-23T03:06:23.9684302Z AssertionError: False is not true
2022-11-23T03:06:23.9684321Z 
2022-11-23T03:06:23.9684339Z 
2022-11-23T03:06:23.9684357Z 
2022-11-23T03:06:23.9684558Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4295
2022-11-23T03:06:23.9684760Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4296
2022-11-23T03:06:23.9685107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9685310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9685848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9686017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9686392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9686573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9686937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9687117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9687354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9687581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9687980Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9688423Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9688811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9689024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9689995Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9690098Z   warnings.warn(
2022-11-23T03:06:23.9691053Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9691154Z   warnings.warn(
2022-11-23T03:06:23.9691340Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9691458Z Traceback (most recent call last):
2022-11-23T03:06:23.9691797Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9691906Z     getattr(self, test_name)()
2022-11-23T03:06:23.9692252Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9692340Z     fn()
2022-11-23T03:06:23.9692681Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9692790Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9693026Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9693126Z     self._dist_train()
2022-11-23T03:06:23.9693338Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9693733Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9693954Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9694091Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9694324Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9694468Z     raise self.failureException(msg)
2022-11-23T03:06:23.9694598Z AssertionError: False is not true
2022-11-23T03:06:23.9694725Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9694909Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9695036Z Traceback (most recent call last):
2022-11-23T03:06:23.9695393Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9695506Z     getattr(self, test_name)()
2022-11-23T03:06:23.9695860Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9695946Z     fn()
2022-11-23T03:06:23.9696293Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9696553Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9696982Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9697335Z     self._dist_train()
2022-11-23T03:06:23.9697562Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9697802Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9698019Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9698150Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9698334Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9698459Z     raise self.failureException(msg)
2022-11-23T03:06:23.9698585Z AssertionError: False is not true
2022-11-23T03:06:23.9698714Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9698816Z dist init r=0, world=2
2022-11-23T03:06:23.9698913Z dist init r=1, world=2
2022-11-23T03:06:23.9699109Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9699205Z ERROR (4.113s)
2022-11-23T03:06:23.9699588Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 7
2022-11-23T03:06:23.9699714Z Traceback (most recent call last):
2022-11-23T03:06:23.9700217Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9700328Z     self._join_processes(fn)
2022-11-23T03:06:23.9700682Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9700814Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9701172Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9701284Z     raise RuntimeError(error)
2022-11-23T03:06:23.9701451Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.9701572Z Traceback (most recent call last):
2022-11-23T03:06:23.9701918Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9702025Z     getattr(self, test_name)()
2022-11-23T03:06:23.9702358Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9702442Z     fn()
2022-11-23T03:06:23.9702777Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9702886Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9703113Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9703211Z     self._dist_train()
2022-11-23T03:06:23.9703432Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9703747Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9703973Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9704268Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9704456Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9704593Z     raise self.failureException(msg)
2022-11-23T03:06:23.9704708Z AssertionError: False is not true
2022-11-23T03:06:23.9704729Z 
2022-11-23T03:06:23.9704886Z Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.9705008Z Traceback (most recent call last):
2022-11-23T03:06:23.9705364Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9705475Z     getattr(self, test_name)()
2022-11-23T03:06:23.9705827Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9705914Z     fn()
2022-11-23T03:06:23.9706255Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9706422Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9706665Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9706768Z     self._dist_train()
2022-11-23T03:06:23.9706995Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9707225Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9707442Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9707569Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9707745Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9707881Z     raise self.failureException(msg)
2022-11-23T03:06:23.9708005Z AssertionError: False is not true
2022-11-23T03:06:23.9708027Z 
2022-11-23T03:06:23.9708045Z 
2022-11-23T03:06:23.9708064Z 
2022-11-23T03:06:23.9708272Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4446
2022-11-23T03:06:23.9708478Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4447
2022-11-23T03:06:23.9708839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9709154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9709520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9709694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9710029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9710191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9710607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9710790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9711013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9711417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9711811Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9712197Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9712415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9712672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9713690Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9713792Z   warnings.warn(
2022-11-23T03:06:23.9714781Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9714931Z   warnings.warn(
2022-11-23T03:06:23.9715124Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9715249Z Traceback (most recent call last):
2022-11-23T03:06:23.9715606Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9715717Z     getattr(self, test_name)()
2022-11-23T03:06:23.9716071Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9716158Z     fn()
2022-11-23T03:06:23.9716499Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9716614Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9716856Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9716956Z     self._dist_train()
2022-11-23T03:06:23.9717186Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9717415Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9717636Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9717757Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9717941Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9718073Z     raise self.failureException(msg)
2022-11-23T03:06:23.9718197Z AssertionError: False is not true
2022-11-23T03:06:23.9718322Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9718512Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9718634Z Traceback (most recent call last):
2022-11-23T03:06:23.9718992Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9719101Z     getattr(self, test_name)()
2022-11-23T03:06:23.9719453Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9719540Z     fn()
2022-11-23T03:06:23.9719887Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9720001Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9720237Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9720335Z     self._dist_train()
2022-11-23T03:06:23.9720560Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9720784Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9721047Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9721185Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9721367Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9721502Z     raise self.failureException(msg)
2022-11-23T03:06:23.9721627Z AssertionError: False is not true
2022-11-23T03:06:23.9721752Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9721845Z dist init r=1, world=2
2022-11-23T03:06:23.9721943Z dist init r=0, world=2
2022-11-23T03:06:23.9722132Z Process 1 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9722227Z ERROR (4.113s)
2022-11-23T03:06:23.9722615Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 6
2022-11-23T03:06:23.9722736Z Traceback (most recent call last):
2022-11-23T03:06:23.9723093Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9723199Z     self._join_processes(fn)
2022-11-23T03:06:23.9723644Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9723782Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9724153Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9724269Z     raise RuntimeError(error)
2022-11-23T03:06:23.9724445Z RuntimeError: Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.9724566Z Traceback (most recent call last):
2022-11-23T03:06:23.9724914Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9725017Z     getattr(self, test_name)()
2022-11-23T03:06:23.9725371Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9725458Z     fn()
2022-11-23T03:06:23.9725805Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9725922Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9726159Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9726258Z     self._dist_train()
2022-11-23T03:06:23.9726482Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9726700Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9726917Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9727044Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9727226Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9727364Z     raise self.failureException(msg)
2022-11-23T03:06:23.9727493Z AssertionError: False is not true
2022-11-23T03:06:23.9727516Z 
2022-11-23T03:06:23.9727534Z 
2022-11-23T03:06:23.9727553Z 
2022-11-23T03:06:23.9727760Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4597
2022-11-23T03:06:23.9727963Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4598
2022-11-23T03:06:23.9728325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9728483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9728852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9729032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9729431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9729601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9729973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9730149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9730386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9730612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9731000Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9731385Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9731608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9731831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9732891Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9732991Z   warnings.warn(
2022-11-23T03:06:23.9733980Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9734081Z   warnings.warn(
2022-11-23T03:06:23.9734272Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9734396Z Traceback (most recent call last):
2022-11-23T03:06:23.9734747Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9734860Z     getattr(self, test_name)()
2022-11-23T03:06:23.9735214Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9735300Z     fn()
2022-11-23T03:06:23.9735651Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9735765Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9736010Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9736110Z     self._dist_train()
2022-11-23T03:06:23.9736333Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9736563Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9736966Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9737105Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9737290Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9737423Z     raise self.failureException(msg)
2022-11-23T03:06:23.9737546Z AssertionError: False is not true
2022-11-23T03:06:23.9737677Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9737861Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9737984Z Traceback (most recent call last):
2022-11-23T03:06:23.9738570Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9738691Z     getattr(self, test_name)()
2022-11-23T03:06:23.9739032Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9739113Z     fn()
2022-11-23T03:06:23.9739444Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9739547Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9739775Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9739872Z     self._dist_train()
2022-11-23T03:06:23.9740090Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9740313Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9740528Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9740714Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9740890Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9741010Z     raise self.failureException(msg)
2022-11-23T03:06:23.9741132Z AssertionError: False is not true
2022-11-23T03:06:23.9741254Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9741350Z dist init r=0, world=2
2022-11-23T03:06:23.9741446Z dist init r=1, world=2
2022-11-23T03:06:23.9741629Z Process 1 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9741895Z ERROR (4.113s)
2022-11-23T03:06:23.9742276Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 5
2022-11-23T03:06:23.9742400Z Traceback (most recent call last):
2022-11-23T03:06:23.9742752Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9742866Z     self._join_processes(fn)
2022-11-23T03:06:23.9743239Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9743373Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9743747Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9743863Z     raise RuntimeError(error)
2022-11-23T03:06:23.9744031Z RuntimeError: Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.9744153Z Traceback (most recent call last):
2022-11-23T03:06:23.9744506Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9744619Z     getattr(self, test_name)()
2022-11-23T03:06:23.9744970Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9745057Z     fn()
2022-11-23T03:06:23.9745560Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9745674Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9745896Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9745991Z     self._dist_train()
2022-11-23T03:06:23.9746208Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9746427Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9746636Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9746758Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9746935Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9747111Z     raise self.failureException(msg)
2022-11-23T03:06:23.9747230Z AssertionError: False is not true
2022-11-23T03:06:23.9747253Z 
2022-11-23T03:06:23.9747271Z 
2022-11-23T03:06:23.9747294Z 
2022-11-23T03:06:23.9747488Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4748
2022-11-23T03:06:23.9747689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4749
2022-11-23T03:06:23.9748044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9748201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9748737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9748917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9749273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9749438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9749846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9750023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9750259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9750491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9750884Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9751270Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9751643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9751852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9752824Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9752924Z   warnings.warn(
2022-11-23T03:06:23.9754052Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9754147Z   warnings.warn(
2022-11-23T03:06:23.9754338Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9754463Z Traceback (most recent call last):
2022-11-23T03:06:23.9754822Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9754933Z     getattr(self, test_name)()
2022-11-23T03:06:23.9755280Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9755367Z     fn()
2022-11-23T03:06:23.9755716Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9755823Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9756110Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9756224Z     self._dist_train()
2022-11-23T03:06:23.9756450Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9756680Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9756900Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9757180Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9757356Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9757478Z     raise self.failureException(msg)
2022-11-23T03:06:23.9757596Z AssertionError: False is not true
2022-11-23T03:06:23.9757719Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9757908Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9758026Z Traceback (most recent call last):
2022-11-23T03:06:23.9758373Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9758528Z     getattr(self, test_name)()
2022-11-23T03:06:23.9758858Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9758942Z     fn()
2022-11-23T03:06:23.9759448Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9759563Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9759803Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9759901Z     self._dist_train()
2022-11-23T03:06:23.9760132Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9760362Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9760577Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9760706Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9760889Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9761021Z     raise self.failureException(msg)
2022-11-23T03:06:23.9761209Z AssertionError: False is not true
2022-11-23T03:06:23.9761334Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9761433Z dist init r=1, world=2
2022-11-23T03:06:23.9761525Z dist init r=0, world=2
2022-11-23T03:06:23.9761715Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9761809Z ERROR (4.013s)
2022-11-23T03:06:23.9762200Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 4
2022-11-23T03:06:23.9762350Z Traceback (most recent call last):
2022-11-23T03:06:23.9762723Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9762837Z     self._join_processes(fn)
2022-11-23T03:06:23.9763207Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9763336Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9763706Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9763820Z     raise RuntimeError(error)
2022-11-23T03:06:23.9763994Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.9764115Z Traceback (most recent call last):
2022-11-23T03:06:23.9764464Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9764576Z     getattr(self, test_name)()
2022-11-23T03:06:23.9764972Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9765057Z     fn()
2022-11-23T03:06:23.9765408Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9765520Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9765763Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9765861Z     self._dist_train()
2022-11-23T03:06:23.9766085Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9766308Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9766518Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9766646Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9766829Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9766967Z     raise self.failureException(msg)
2022-11-23T03:06:23.9767093Z AssertionError: False is not true
2022-11-23T03:06:23.9767159Z 
2022-11-23T03:06:23.9767177Z 
2022-11-23T03:06:23.9767196Z 
2022-11-23T03:06:23.9767408Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4899
2022-11-23T03:06:23.9767613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4900
2022-11-23T03:06:23.9767976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9768293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9768645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9768818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9769163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9769322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9769674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9769845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9770073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9770297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9770672Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9771208Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9771431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9771646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9772656Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9772760Z   warnings.warn(
2022-11-23T03:06:23.9773792Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9773899Z   warnings.warn(
2022-11-23T03:06:23.9774243Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9774364Z Traceback (most recent call last):
2022-11-23T03:06:23.9774708Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9774817Z     getattr(self, test_name)()
2022-11-23T03:06:23.9775151Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9775234Z     fn()
2022-11-23T03:06:23.9775571Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9775684Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9775922Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9776065Z     self._dist_train()
2022-11-23T03:06:23.9776283Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9776506Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9777061Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9777202Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9777389Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9777520Z     raise self.failureException(msg)
2022-11-23T03:06:23.9777643Z AssertionError: False is not true
2022-11-23T03:06:23.9777768Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9777959Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9778074Z Traceback (most recent call last):
2022-11-23T03:06:23.9778441Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9778555Z     getattr(self, test_name)()
2022-11-23T03:06:23.9778903Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9778988Z     fn()
2022-11-23T03:06:23.9779337Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9779448Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9779843Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9779932Z     self._dist_train()
2022-11-23T03:06:23.9780156Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9780373Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9780585Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9780711Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9781060Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9781192Z     raise self.failureException(msg)
2022-11-23T03:06:23.9781309Z AssertionError: False is not true
2022-11-23T03:06:23.9781437Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9781535Z dist init r=1, world=2
2022-11-23T03:06:23.9781634Z dist init r=0, world=2
2022-11-23T03:06:23.9781823Z Process 1 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9781916Z ERROR (4.013s)
2022-11-23T03:06:23.9782304Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 3
2022-11-23T03:06:23.9782427Z Traceback (most recent call last):
2022-11-23T03:06:23.9782843Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9782968Z     self._join_processes(fn)
2022-11-23T03:06:23.9783337Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9783472Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9783993Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9784103Z     raise RuntimeError(error)
2022-11-23T03:06:23.9784271Z RuntimeError: Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.9784382Z Traceback (most recent call last):
2022-11-23T03:06:23.9784721Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9784830Z     getattr(self, test_name)()
2022-11-23T03:06:23.9785163Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9785245Z     fn()
2022-11-23T03:06:23.9785843Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9785955Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9786191Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9786283Z     self._dist_train()
2022-11-23T03:06:23.9786514Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9786740Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9786962Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9787089Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9787281Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9787415Z     raise self.failureException(msg)
2022-11-23T03:06:23.9787540Z AssertionError: False is not true
2022-11-23T03:06:23.9787561Z 
2022-11-23T03:06:23.9787580Z 
2022-11-23T03:06:23.9787598Z 
2022-11-23T03:06:23.9787807Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5050
2022-11-23T03:06:23.9788007Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5051
2022-11-23T03:06:23.9788376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9788538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9788910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9789250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9789591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9789749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9790104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9790268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9790492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9790718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9791097Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9791469Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9791728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9791944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9792923Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9793024Z   warnings.warn(
2022-11-23T03:06:23.9794153Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9794298Z   warnings.warn(
2022-11-23T03:06:23.9794490Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9794606Z Traceback (most recent call last):
2022-11-23T03:06:23.9794965Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9795076Z     getattr(self, test_name)()
2022-11-23T03:06:23.9795428Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9795512Z     fn()
2022-11-23T03:06:23.9795861Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9795978Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9796214Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9796317Z     self._dist_train()
2022-11-23T03:06:23.9796546Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9796928Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9797138Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9797263Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9797611Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9797744Z     raise self.failureException(msg)
2022-11-23T03:06:23.9797862Z AssertionError: False is not true
2022-11-23T03:06:23.9797987Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9798176Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9798303Z Traceback (most recent call last):
2022-11-23T03:06:23.9798661Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9798779Z     getattr(self, test_name)()
2022-11-23T03:06:23.9799127Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9799206Z     fn()
2022-11-23T03:06:23.9799552Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9799665Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9799906Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9800007Z     self._dist_train()
2022-11-23T03:06:23.9800232Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9800661Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9800877Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9800997Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9801173Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9801301Z     raise self.failureException(msg)
2022-11-23T03:06:23.9801420Z AssertionError: False is not true
2022-11-23T03:06:23.9801541Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9801638Z dist init r=1, world=2
2022-11-23T03:06:23.9801734Z dist init r=0, world=2
2022-11-23T03:06:23.9801911Z Process 1 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9802001Z ERROR (4.013s)
2022-11-23T03:06:23.9802374Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 2
2022-11-23T03:06:23.9802492Z Traceback (most recent call last):
2022-11-23T03:06:23.9802835Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9802994Z     self._join_processes(fn)
2022-11-23T03:06:23.9803352Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9803481Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9803834Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9803947Z     raise RuntimeError(error)
2022-11-23T03:06:23.9804117Z RuntimeError: Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.9804236Z Traceback (most recent call last):
2022-11-23T03:06:23.9804573Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9804680Z     getattr(self, test_name)()
2022-11-23T03:06:23.9805017Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9805103Z     fn()
2022-11-23T03:06:23.9805431Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9805540Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9805769Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9805864Z     self._dist_train()
2022-11-23T03:06:23.9806080Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9806298Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9806510Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9806632Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9806804Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9806933Z     raise self.failureException(msg)
2022-11-23T03:06:23.9807053Z AssertionError: False is not true
2022-11-23T03:06:23.9807072Z 
2022-11-23T03:06:23.9807091Z 
2022-11-23T03:06:23.9807109Z 
2022-11-23T03:06:23.9807307Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5201
2022-11-23T03:06:23.9807503Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5202
2022-11-23T03:06:23.9807855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9808014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9808369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9808542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9808923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9809091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9809446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9809615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9809843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9810071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9810688Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9811074Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9811296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9811576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9812584Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9812685Z   warnings.warn(
2022-11-23T03:06:23.9813675Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9813776Z   warnings.warn(
2022-11-23T03:06:23.9813966Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9814088Z Traceback (most recent call last):
2022-11-23T03:06:23.9814442Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9814555Z     getattr(self, test_name)()
2022-11-23T03:06:23.9814908Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9814988Z     fn()
2022-11-23T03:06:23.9815336Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9815448Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9815690Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9815792Z     self._dist_train()
2022-11-23T03:06:23.9816017Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9816244Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9816462Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9816583Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9816955Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9817095Z     raise self.failureException(msg)
2022-11-23T03:06:23.9817219Z AssertionError: False is not true
2022-11-23T03:06:23.9817344Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9817444Z dist init r=1, world=2
2022-11-23T03:06:23.9817706Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9817829Z Traceback (most recent call last):
2022-11-23T03:06:23.9818350Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9818458Z     getattr(self, test_name)()
2022-11-23T03:06:23.9818796Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9818878Z     fn()
2022-11-23T03:06:23.9819210Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9819318Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9819549Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9819638Z     self._dist_train()
2022-11-23T03:06:23.9819859Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9820081Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9820359Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9820481Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9820659Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9820787Z     raise self.failureException(msg)
2022-11-23T03:06:23.9820900Z AssertionError: False is not true
2022-11-23T03:06:23.9821019Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9821115Z dist init r=0, world=2
2022-11-23T03:06:23.9821300Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9821389Z ERROR (4.113s)
2022-11-23T03:06:23.9821762Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ...     test_forward_overlap errored - num_retries_left: 1
2022-11-23T03:06:23.9821880Z Traceback (most recent call last):
2022-11-23T03:06:23.9822220Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 532, in wrapper
2022-11-23T03:06:23.9822325Z     self._join_processes(fn)
2022-11-23T03:06:23.9822678Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 758, in _join_processes
2022-11-23T03:06:23.9822806Z     self._check_return_codes(elapsed_time)
2022-11-23T03:06:23.9823164Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 803, in _check_return_codes
2022-11-23T03:06:23.9823274Z     raise RuntimeError(error)
2022-11-23T03:06:23.9823612Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-11-23T03:06:23.9823735Z Traceback (most recent call last):
2022-11-23T03:06:23.9824084Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9824189Z     getattr(self, test_name)()
2022-11-23T03:06:23.9824547Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9824636Z     fn()
2022-11-23T03:06:23.9824980Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9825091Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9825326Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9825425Z     self._dist_train()
2022-11-23T03:06:23.9825649Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9825870Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9826085Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9826211Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9826440Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9826578Z     raise self.failureException(msg)
2022-11-23T03:06:23.9826704Z AssertionError: False is not true
2022-11-23T03:06:23.9826725Z 
2022-11-23T03:06:23.9826875Z Process 1 exited with error code 10 and exception:
2022-11-23T03:06:23.9826990Z Traceback (most recent call last):
2022-11-23T03:06:23.9827345Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9827457Z     getattr(self, test_name)()
2022-11-23T03:06:23.9827804Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9827888Z     fn()
2022-11-23T03:06:23.9828231Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9828344Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9828587Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9828680Z     self._dist_train()
2022-11-23T03:06:23.9828952Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9829177Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9829547Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9829672Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9829847Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9829975Z     raise self.failureException(msg)
2022-11-23T03:06:23.9830093Z AssertionError: False is not true
2022-11-23T03:06:23.9830112Z 
2022-11-23T03:06:23.9830130Z 
2022-11-23T03:06:23.9830148Z 
2022-11-23T03:06:23.9830345Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5352
2022-11-23T03:06:23.9830539Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5353
2022-11-23T03:06:23.9830893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9831054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9831407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9831750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9832102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:06:23.9832261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:06:23.9832621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:06:23.9832791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:06:23.9833027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:06:23.9833262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:06:23.9833652Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9834035Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T03:06:23.9834256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:06:23.9834470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:06:23.9835518Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9835629Z   warnings.warn(
2022-11-23T03:06:23.9836616Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:06:23.9836714Z   warnings.warn(
2022-11-23T03:06:23.9836899Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9837022Z Traceback (most recent call last):
2022-11-23T03:06:23.9837377Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9837575Z     getattr(self, test_name)()
2022-11-23T03:06:23.9837931Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9838016Z     fn()
2022-11-23T03:06:23.9838362Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9838474Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9838707Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9838807Z     self._dist_train()
2022-11-23T03:06:23.9839033Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9839259Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9839479Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9839611Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9839794Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9839926Z     raise self.failureException(msg)
2022-11-23T03:06:23.9840043Z AssertionError: False is not true
2022-11-23T03:06:23.9840168Z  exiting process 0 with exit code: 10
2022-11-23T03:06:23.9840265Z dist init r=0, world=2
2022-11-23T03:06:23.9840456Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-11-23T03:06:23.9840577Z Traceback (most recent call last):
2022-11-23T03:06:23.9840934Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test
2022-11-23T03:06:23.9841044Z     getattr(self, test_name)()
2022-11-23T03:06:23.9841392Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper
2022-11-23T03:06:23.9841478Z     fn()
2022-11-23T03:06:23.9841829Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper
2022-11-23T03:06:23.9841943Z     return func(*args, **kwargs)
2022-11-23T03:06:23.9842184Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 240, in test_forward_overlap
2022-11-23T03:06:23.9842282Z     self._dist_train()
2022-11-23T03:06:23.9842508Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 186, in _dist_train
2022-11-23T03:06:23.9842735Z     e2 = run(0, sleep_cycles)  # no compute, only all-gather
2022-11-23T03:06:23.9842945Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_overlap.py", line 145, in run
2022-11-23T03:06:23.9843073Z     self.assertTrue(all_gather_called)
2022-11-23T03:06:23.9843255Z   File "/opt/conda/lib/python3.10/unittest/case.py", line 687, in assertTrue
2022-11-23T03:06:23.9843435Z     raise self.failureException(msg)
2022-11-23T03:06:23.9843564Z AssertionError: False is not true
2022-11-23T03:06:23.9843692Z  exiting process 1 with exit code: 10
2022-11-23T03:06:23.9843790Z dist init r=1, world=2
2022-11-23T03:06:23.9843974Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-11-23T03:06:23.9844067Z ERROR (4.013s)
2022-11-23T03:06:23.9844285Z     test_forward_overlap errored - num_retries_left: 0
2022-11-23T03:06:23.9844305Z 
2022-11-23T03:06:23.9844563Z ----------------------------------------------------------------------
2022-11-23T03:06:23.9844672Z Ran 52 tests in 210.898s
2022-11-23T03:06:23.9844691Z 
2022-11-23T03:06:23.9844972Z OK (skipped=2, expected failures=50)
2022-11-23T03:06:23.9844991Z 
2022-11-23T03:06:23.9845097Z Generating XML reports...
2022-11-23T03:06:23.9845751Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20221123030252.xml
2022-11-23T03:06:23.9846239Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20221123030252.xml
2022-11-23T03:06:23.9846317Z 
2022-11-23T03:06:23.9846734Z ##[endgroup]
2022-11-23T03:06:23.9847199Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_overlap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_overlap_ccye88j5)
2022-11-23T03:06:23.9847219Z 
2022-11-23T03:06:24.1797583Z 
2022-11-23T03:06:24.1798185Z real	3m39.010s
2022-11-23T03:06:24.1798324Z user	7m31.918s
2022-11-23T03:06:24.1798930Z sys	6m8.699s
2022-11-23T03:06:24.1799170Z + for f in test/distributed/fsdp/*.py
2022-11-23T03:06:24.1799526Z + python test/run_test.py --verbose -i distributed/fsdp/test_fsdp_pure_fp16.py
2022-11-23T03:06:26.5819763Z Ignoring disabled issues:  []
2022-11-23T03:06:26.6348357Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:06:26.6349007Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:06:26.6349228Z Selected tests:
2022-11-23T03:06:26.6349548Z  distributed/fsdp/test_fsdp_pure_fp16.py
2022-11-23T03:06:26.6376071Z Prioritized test from test file changes.
2022-11-23T03:06:26.6376778Z reordering tests for PR:
2022-11-23T03:06:26.6377125Z prioritized: []
2022-11-23T03:06:26.6378037Z the rest: ['distributed/fsdp/test_fsdp_pure_fp16.py']
2022-11-23T03:06:26.6378156Z 
2022-11-23T03:06:26.6378818Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:06:26.6379635Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:06:26.6385777Z parallel (file granularity) tests:
2022-11-23T03:06:26.6386422Z  
2022-11-23T03:06:26.6386798Z serial (file granularity) tests:
2022-11-23T03:06:26.6387069Z  distributed/fsdp/test_fsdp_pure_fp16.py
2022-11-23T03:06:28.8930079Z Ignoring disabled issues:  []
2022-11-23T03:06:28.8991295Z Ignoring disabled issues:  []
2022-11-23T03:06:29.3463717Z Running distributed/fsdp/test_fsdp_pure_fp16.py ... [2022-11-23 03:06:29.345866]
2022-11-23T03:06:29.3466447Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_pure_fp16.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:06:29.346310]
2022-11-23T03:10:41.9528206Z 
2022-11-23T03:10:41.9528792Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_pure_fp16
2022-11-23T03:10:41.9530602Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_pure_fp16 (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_pure_fp16_vfceuz13)
2022-11-23T03:10:41.9532224Z 
2022-11-23T03:10:41.9532561Z Running tests...
2022-11-23T03:10:41.9533838Z ----------------------------------------------------------------------
2022-11-23T03:10:41.9537048Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_pure_fp16
2022-11-23T03:10:41.9537601Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:41.9538215Z Tests pure FP16 training, including when the parameter's dtype is ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:10:41.9538714Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5785
2022-11-23T03:10:41.9539150Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5786
2022-11-23T03:10:41.9539616Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5787
2022-11-23T03:10:41.9540063Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5788
2022-11-23T03:10:41.9540677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9541122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9541714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9542397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9542998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9543435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9544004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9544479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9545064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9545493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9546084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9546552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9547113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9547573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9548164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9548630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9549074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:41.9549582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:41.9550088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:41.9550591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:41.9551241Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9551959Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9552667Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9553349Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9553851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:41.9554435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:41.9554921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:41.9555377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:41.9556692Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9557560Z   warnings.warn(
2022-11-23T03:10:41.9558729Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9559587Z   warnings.warn(
2022-11-23T03:10:41.9560749Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9561537Z   warnings.warn(
2022-11-23T03:10:41.9562680Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9563457Z   warnings.warn(
2022-11-23T03:10:41.9563692Z dist init r=2, world=4
2022-11-23T03:10:41.9563950Z dist init r=3, world=4
2022-11-23T03:10:41.9564201Z dist init r=1, world=4
2022-11-23T03:10:41.9564422Z dist init r=0, world=4
2022-11-23T03:10:41.9564676Z ok (6.660s)
2022-11-23T03:10:41.9565026Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:41.9565738Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 50
2022-11-23T03:10:41.9566307Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6086
2022-11-23T03:10:41.9566760Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6087
2022-11-23T03:10:41.9567200Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6088
2022-11-23T03:10:41.9567625Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6089
2022-11-23T03:10:41.9568238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9568699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9569264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9569739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9570319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9570827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9571407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9571883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9572469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9573354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9573934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9574403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9574984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9575425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9576009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9576564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9577427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:41.9577917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:41.9578417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:41.9578903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:41.9579564Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9580251Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9580961Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9581660Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9582187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:41.9582724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:41.9583207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:41.9583685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:41.9584973Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9585764Z   warnings.warn(
2022-11-23T03:10:41.9586897Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9587662Z   warnings.warn(
2022-11-23T03:10:41.9588909Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9589689Z   warnings.warn(
2022-11-23T03:10:41.9590854Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9591608Z   warnings.warn(
2022-11-23T03:10:41.9591845Z dist init r=1, world=4
2022-11-23T03:10:41.9592092Z dist init r=0, world=4
2022-11-23T03:10:41.9592355Z dist init r=3, world=4
2022-11-23T03:10:41.9592672Z dist init r=2, world=4
2022-11-23T03:10:41.9592908Z ok (4.818s)
2022-11-23T03:10:41.9593254Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:41.9593954Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 49
2022-11-23T03:10:41.9594506Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6387
2022-11-23T03:10:41.9594958Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6388
2022-11-23T03:10:41.9595402Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6389
2022-11-23T03:10:41.9595814Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6390
2022-11-23T03:10:41.9596422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9596878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9597456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9597910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9598486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9598931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9599484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9599949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9600526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9600968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9601525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9601985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9602556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9602979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9603560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9604028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9604486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:41.9605028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:41.9605535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:41.9606039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:41.9606707Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9607385Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9608074Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9608771Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9609298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:41.9609760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:41.9610288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:41.9610766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:41.9612018Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9612806Z   warnings.warn(
2022-11-23T03:10:41.9613968Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9614742Z   warnings.warn(
2022-11-23T03:10:41.9615890Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9616665Z   warnings.warn(
2022-11-23T03:10:41.9618017Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9618798Z   warnings.warn(
2022-11-23T03:10:41.9619058Z dist init r=1, world=4
2022-11-23T03:10:41.9619316Z dist init r=0, world=4
2022-11-23T03:10:41.9619547Z dist init r=3, world=4
2022-11-23T03:10:41.9619800Z dist init r=2, world=4
2022-11-23T03:10:41.9620039Z ok (4.818s)
2022-11-23T03:10:41.9620375Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:41.9621173Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 48
2022-11-23T03:10:41.9621742Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6688
2022-11-23T03:10:41.9622202Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6689
2022-11-23T03:10:41.9622631Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6690
2022-11-23T03:10:41.9623072Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6691
2022-11-23T03:10:41.9623694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9624203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9624998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9625451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9626038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9626496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9627201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9627673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9628258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9628686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9629258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9629730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9630296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9630745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9631325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9631792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9632230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:41.9632733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:41.9633226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:41.9633699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:41.9634363Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9635068Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9635770Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9636435Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9636954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:41.9637433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:41.9637904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:41.9638353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:41.9639677Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9640474Z   warnings.warn(
2022-11-23T03:10:41.9641633Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9642404Z   warnings.warn(
2022-11-23T03:10:41.9643532Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9644356Z   warnings.warn(
2022-11-23T03:10:41.9645516Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9646290Z   warnings.warn(
2022-11-23T03:10:41.9646541Z dist init r=3, world=4
2022-11-23T03:10:41.9646775Z dist init r=1, world=4
2022-11-23T03:10:41.9647030Z dist init r=0, world=4
2022-11-23T03:10:41.9647284Z dist init r=2, world=4
2022-11-23T03:10:41.9647503Z ok (4.818s)
2022-11-23T03:10:41.9647856Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:41.9648567Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 47
2022-11-23T03:10:41.9649123Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6989
2022-11-23T03:10:41.9649556Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6990
2022-11-23T03:10:41.9649997Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6991
2022-11-23T03:10:41.9650441Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6992
2022-11-23T03:10:41.9651039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9651503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9652084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9652565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9653132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9653587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9654166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9654616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9655263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9655727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9656305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9657021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9657618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9658066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9677226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9677787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9678269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:41.9678760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:41.9679420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:41.9679919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:41.9680619Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9681295Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9681981Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9682754Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9683278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:41.9683737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:41.9684204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:41.9684671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:41.9685944Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9686703Z   warnings.warn(
2022-11-23T03:10:41.9687843Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9688615Z   warnings.warn(
2022-11-23T03:10:41.9689751Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9690605Z   warnings.warn(
2022-11-23T03:10:41.9691759Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9692496Z   warnings.warn(
2022-11-23T03:10:41.9692744Z dist init r=3, world=4
2022-11-23T03:10:41.9693002Z dist init r=1, world=4
2022-11-23T03:10:41.9693231Z dist init r=0, world=4
2022-11-23T03:10:41.9693474Z dist init r=2, world=4
2022-11-23T03:10:41.9693712Z ok (4.918s)
2022-11-23T03:10:41.9694040Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:41.9694748Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 46
2022-11-23T03:10:41.9695375Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7290
2022-11-23T03:10:41.9695839Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7291
2022-11-23T03:10:41.9696274Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7292
2022-11-23T03:10:41.9696961Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7293
2022-11-23T03:10:41.9697609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9698081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9698652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9699140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9699724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9700160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9700742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9701275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9701871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9702301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9702880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9703354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9703926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9704386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9704958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9705430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9705869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:41.9706383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:41.9706890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:41.9707393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:41.9708184Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9708911Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9709605Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9710272Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9710800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:41.9711281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:41.9711754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:41.9712211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:41.9713479Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9714352Z   warnings.warn(
2022-11-23T03:10:41.9715509Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9716296Z   warnings.warn(
2022-11-23T03:10:41.9717446Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9718186Z   warnings.warn(
2022-11-23T03:10:41.9719342Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9720105Z   warnings.warn(
2022-11-23T03:10:41.9720366Z dist init r=1, world=4
2022-11-23T03:10:41.9720599Z dist init r=3, world=4
2022-11-23T03:10:41.9720858Z dist init r=2, world=4
2022-11-23T03:10:41.9721112Z dist init r=0, world=4
2022-11-23T03:10:41.9721332Z ok (4.918s)
2022-11-23T03:10:41.9721686Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:41.9722404Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 45
2022-11-23T03:10:41.9722960Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7591
2022-11-23T03:10:41.9723396Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7592
2022-11-23T03:10:41.9723847Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7593
2022-11-23T03:10:41.9724442Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7594
2022-11-23T03:10:41.9725062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9725530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9726116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9726596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9727162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9727616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9728195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9728678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9729245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9729769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9730354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9730800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9731392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9731836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9732409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9732855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9733319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:41.9733829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:41.9734312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:41.9734811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:41.9735471Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9736169Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9737054Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9737763Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9738297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:41.9738780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:41.9739238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:41.9739710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:41.9741064Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9741862Z   warnings.warn(
2022-11-23T03:10:41.9743019Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9743774Z   warnings.warn(
2022-11-23T03:10:41.9744922Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9745778Z   warnings.warn(
2022-11-23T03:10:41.9746918Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9747694Z   warnings.warn(
2022-11-23T03:10:41.9747930Z dist init r=1, world=4
2022-11-23T03:10:41.9748186Z dist init r=3, world=4
2022-11-23T03:10:41.9748437Z dist init r=2, world=4
2022-11-23T03:10:41.9748669Z dist init r=0, world=4
2022-11-23T03:10:41.9748914Z ok (4.818s)
2022-11-23T03:10:41.9749279Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:41.9749996Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 44
2022-11-23T03:10:41.9750539Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7892
2022-11-23T03:10:41.9751002Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7893
2022-11-23T03:10:41.9751462Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7894
2022-11-23T03:10:41.9751893Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7895
2022-11-23T03:10:41.9752503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9752969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9753558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9754019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9754612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9755064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9755622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9756095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9756665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9757113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9757664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9758197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9758796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9759254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9759806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9760274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9760733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:41.9761220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:41.9761736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:41.9762242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:41.9762906Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9763648Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9764345Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9765031Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9765556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:41.9766016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:41.9766497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:41.9766971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:41.9768245Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9769005Z   warnings.warn(
2022-11-23T03:10:41.9770157Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9770922Z   warnings.warn(
2022-11-23T03:10:41.9772058Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9772831Z   warnings.warn(
2022-11-23T03:10:41.9774030Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9774790Z   warnings.warn(
2022-11-23T03:10:41.9775049Z dist init r=1, world=4
2022-11-23T03:10:41.9775307Z dist init r=2, world=4
2022-11-23T03:10:41.9775537Z dist init r=3, world=4
2022-11-23T03:10:41.9775789Z dist init r=0, world=4
2022-11-23T03:10:41.9776031Z ok (4.918s)
2022-11-23T03:10:41.9776363Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:41.9777333Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 43
2022-11-23T03:10:41.9777896Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8193
2022-11-23T03:10:41.9778356Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8194
2022-11-23T03:10:41.9778796Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8195
2022-11-23T03:10:41.9779245Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8196
2022-11-23T03:10:41.9779974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9780450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9781011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9781492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9782071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9782562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9783154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9783635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9784223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9784648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9785228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9785700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9786258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9786703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9787275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9787892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9788506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:41.9789011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:41.9789517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:41.9790031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:41.9790680Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9791377Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9792294Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9792982Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9793472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:41.9793942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:41.9794401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:41.9794835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:41.9796282Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9797140Z   warnings.warn(
2022-11-23T03:10:41.9798443Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9799199Z   warnings.warn(
2022-11-23T03:10:41.9800307Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9801027Z   warnings.warn(
2022-11-23T03:10:41.9802125Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9802866Z   warnings.warn(
2022-11-23T03:10:41.9803125Z dist init r=1, world=4
2022-11-23T03:10:41.9803353Z dist init r=0, world=4
2022-11-23T03:10:41.9803778Z dist init r=3, world=4
2022-11-23T03:10:41.9804032Z dist init r=2, world=4
2022-11-23T03:10:41.9804309Z ok (4.919s)
2022-11-23T03:10:41.9804674Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:41.9805396Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 42
2022-11-23T03:10:41.9805954Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8494
2022-11-23T03:10:41.9806392Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8495
2022-11-23T03:10:41.9806847Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8496
2022-11-23T03:10:41.9807303Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8497
2022-11-23T03:10:41.9807923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9808361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9809009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9809659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9810211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9810654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9811394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9811867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9812434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9812884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9813465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9813917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9814564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9815016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9815594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9816041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9816503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:41.9817220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:41.9817746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:41.9818228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:41.9818904Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9819610Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9820426Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9821090Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9821597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:41.9822060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:41.9822503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:41.9822959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:41.9824429Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9825226Z   warnings.warn(
2022-11-23T03:10:41.9826475Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9827252Z   warnings.warn(
2022-11-23T03:10:41.9828525Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9829265Z   warnings.warn(
2022-11-23T03:10:41.9830382Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9831211Z   warnings.warn(
2022-11-23T03:10:41.9831438Z dist init r=3, world=4
2022-11-23T03:10:41.9831691Z dist init r=1, world=4
2022-11-23T03:10:41.9831944Z dist init r=0, world=4
2022-11-23T03:10:41.9832170Z dist init r=2, world=4
2022-11-23T03:10:41.9832587Z ok (4.919s)
2022-11-23T03:10:41.9832946Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:41.9833658Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 41
2022-11-23T03:10:41.9834191Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8795
2022-11-23T03:10:41.9834663Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8796
2022-11-23T03:10:41.9835119Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8797
2022-11-23T03:10:41.9835555Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8798
2022-11-23T03:10:41.9836174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9836637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9837220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9837679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9838263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9838713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9839293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9839753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9840336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9840788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9841345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9841817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9842400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9842847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9843459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9843946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9844410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:41.9844900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:41.9845402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:41.9845900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:41.9846713Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9847367Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9848235Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9848999Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9849527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:41.9849985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:41.9850466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:41.9851096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:41.9852330Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9853071Z   warnings.warn(
2022-11-23T03:10:41.9854185Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9854938Z   warnings.warn(
2022-11-23T03:10:41.9856229Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9857284Z   warnings.warn(
2022-11-23T03:10:41.9858443Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9859197Z   warnings.warn(
2022-11-23T03:10:41.9859456Z dist init r=3, world=4
2022-11-23T03:10:41.9859712Z dist init r=2, world=4
2022-11-23T03:10:41.9860103Z dist init r=1, world=4
2022-11-23T03:10:41.9860433Z dist init r=0, world=4
2022-11-23T03:10:41.9860688Z ok (4.919s)
2022-11-23T03:10:41.9861044Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:41.9861714Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 40
2022-11-23T03:10:41.9862418Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9096
2022-11-23T03:10:41.9862872Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9097
2022-11-23T03:10:41.9863299Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9098
2022-11-23T03:10:41.9863741Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9099
2022-11-23T03:10:41.9864343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9864796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9865358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9865931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9866510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9866936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9867509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9867966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9868534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9868956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9869525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9869987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9870569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9870990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9871554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9872015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9872449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:41.9872953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:41.9873603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:41.9874088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:41.9874700Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9875363Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9876021Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9876869Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9877364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:41.9877895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:41.9878373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:41.9878827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:41.9880222Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9880975Z   warnings.warn(
2022-11-23T03:10:41.9882080Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9883121Z   warnings.warn(
2022-11-23T03:10:41.9884263Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9885015Z   warnings.warn(
2022-11-23T03:10:41.9886152Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9887067Z   warnings.warn(
2022-11-23T03:10:41.9887306Z dist init r=0, world=4
2022-11-23T03:10:41.9887530Z dist init r=1, world=4
2022-11-23T03:10:41.9887767Z dist init r=3, world=4
2022-11-23T03:10:41.9888003Z dist init r=2, world=4
2022-11-23T03:10:41.9888226Z ok (4.919s)
2022-11-23T03:10:41.9888728Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:41.9889434Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 39
2022-11-23T03:10:41.9889983Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9397
2022-11-23T03:10:41.9890413Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9398
2022-11-23T03:10:41.9890860Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9399
2022-11-23T03:10:41.9891460Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9400
2022-11-23T03:10:41.9892048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9892466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9893022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9893476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9894019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9894507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9895079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9895720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9896283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9896933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9897541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9897992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9898577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9899029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9899594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9900290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9900909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:41.9901409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:41.9901905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:41.9902380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:41.9903038Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9903734Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9904424Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9905082Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9905604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:41.9906069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:41.9906519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:41.9906983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:41.9908241Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9909022Z   warnings.warn(
2022-11-23T03:10:41.9910303Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9911032Z   warnings.warn(
2022-11-23T03:10:41.9912377Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9913163Z   warnings.warn(
2022-11-23T03:10:41.9914294Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9915054Z   warnings.warn(
2022-11-23T03:10:41.9915303Z dist init r=2, world=4
2022-11-23T03:10:41.9915538Z dist init r=0, world=4
2022-11-23T03:10:41.9915785Z dist init r=1, world=4
2022-11-23T03:10:41.9916092Z dist init r=3, world=4
2022-11-23T03:10:41.9916307Z ok (4.818s)
2022-11-23T03:10:41.9916654Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:41.9917506Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 38
2022-11-23T03:10:41.9918017Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9698
2022-11-23T03:10:41.9918456Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9699
2022-11-23T03:10:41.9918891Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9700
2022-11-23T03:10:41.9919319Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9701
2022-11-23T03:10:41.9919892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9920525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9921111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9921569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9922156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9922606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9923324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9923762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9924566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9925021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9925581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9926052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9926639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9927089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9927638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9928259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9928709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:41.9929260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:41.9929734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:41.9930227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:41.9930877Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9931523Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9932196Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9933040Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9933572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:41.9934030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:41.9934563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:41.9935042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:41.9936630Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9937622Z   warnings.warn(
2022-11-23T03:10:41.9938762Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9939708Z   warnings.warn(
2022-11-23T03:10:41.9941019Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9941794Z   warnings.warn(
2022-11-23T03:10:41.9942934Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9943712Z   warnings.warn(
2022-11-23T03:10:41.9943946Z dist init r=1, world=4
2022-11-23T03:10:41.9944208Z dist init r=0, world=4
2022-11-23T03:10:41.9944466Z dist init r=3, world=4
2022-11-23T03:10:41.9944696Z dist init r=2, world=4
2022-11-23T03:10:41.9944940Z ok (4.918s)
2022-11-23T03:10:41.9945297Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:41.9946080Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 37
2022-11-23T03:10:41.9946806Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9999
2022-11-23T03:10:41.9947258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10000
2022-11-23T03:10:41.9947685Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10001
2022-11-23T03:10:41.9948310Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10002
2022-11-23T03:10:41.9948937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9949402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9949962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9950444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9951197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9951712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9952257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9952718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9953280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9953693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9954250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9954704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9955270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9955680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9956418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9956881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9957320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:41.9957827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:41.9958330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:41.9958828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:41.9959619Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9960297Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9960971Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9961799Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9962299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:41.9962775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:41.9963254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:41.9963725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:41.9965029Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9965822Z   warnings.warn(
2022-11-23T03:10:41.9967123Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9967876Z   warnings.warn(
2022-11-23T03:10:41.9969167Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9970008Z   warnings.warn(
2022-11-23T03:10:41.9971129Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9971898Z   warnings.warn(
2022-11-23T03:10:41.9972158Z dist init r=3, world=4
2022-11-23T03:10:41.9972418Z dist init r=0, world=4
2022-11-23T03:10:41.9972653Z dist init r=2, world=4
2022-11-23T03:10:41.9972904Z dist init r=1, world=4
2022-11-23T03:10:41.9973300Z ok (4.918s)
2022-11-23T03:10:41.9973625Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:41.9974315Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 36
2022-11-23T03:10:41.9974860Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10300
2022-11-23T03:10:41.9975282Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10301
2022-11-23T03:10:41.9975728Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10302
2022-11-23T03:10:41.9976170Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10303
2022-11-23T03:10:41.9977139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9977592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9978195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9978682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9979249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9979704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9980283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9980753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9981396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9981875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9982673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9983305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9983872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:41.9984322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:41.9984892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:41.9985342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:41.9985806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:41.9986315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:41.9986922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:41.9987398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:41.9988065Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9988766Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9989460Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9990135Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:41.9990667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:41.9991145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:41.9991758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:41.9992216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:41.9993440Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9994205Z   warnings.warn(
2022-11-23T03:10:41.9995317Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9996253Z   warnings.warn(
2022-11-23T03:10:41.9997417Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:41.9998211Z   warnings.warn(
2022-11-23T03:10:41.9999501Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0000249Z   warnings.warn(
2022-11-23T03:10:42.0000498Z dist init r=3, world=4
2022-11-23T03:10:42.0000725Z dist init r=0, world=4
2022-11-23T03:10:42.0000973Z dist init r=2, world=4
2022-11-23T03:10:42.0001513Z dist init r=1, world=4
2022-11-23T03:10:42.0001735Z ok (4.818s)
2022-11-23T03:10:42.0002097Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0002823Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 35
2022-11-23T03:10:42.0003425Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10601
2022-11-23T03:10:42.0003895Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10602
2022-11-23T03:10:42.0004357Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10603
2022-11-23T03:10:42.0004813Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10604
2022-11-23T03:10:42.0005415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0005870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0006461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0006918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0007505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0007958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0008532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0008983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0009566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0010167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0010890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0011360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0011951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0012404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0012959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0013428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0013887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0014395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0014875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0015371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0016086Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0017065Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0017784Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0018475Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0019006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0019462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0019940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0020410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0021843Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0022697Z   warnings.warn(
2022-11-23T03:10:42.0023795Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0024801Z   warnings.warn(
2022-11-23T03:10:42.0025948Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0026728Z   warnings.warn(
2022-11-23T03:10:42.0027878Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0028794Z   warnings.warn(
2022-11-23T03:10:42.0029019Z dist init r=1, world=4
2022-11-23T03:10:42.0029265Z dist init r=3, world=4
2022-11-23T03:10:42.0029510Z dist init r=2, world=4
2022-11-23T03:10:42.0029733Z dist init r=0, world=4
2022-11-23T03:10:42.0029966Z ok (4.918s)
2022-11-23T03:10:42.0030308Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0030977Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 34
2022-11-23T03:10:42.0031517Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10902
2022-11-23T03:10:42.0031968Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10903
2022-11-23T03:10:42.0032487Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10904
2022-11-23T03:10:42.0032923Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10905
2022-11-23T03:10:42.0033727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0034187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0034748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0035222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0035812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0036419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0037140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0037616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0038199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0038704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0039283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0039908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0040476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0040888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0041444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0041905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0042331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0042997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0043504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0044003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0044646Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0045351Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0046051Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0046902Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0047391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0047860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0048497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0048971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0050276Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0051065Z   warnings.warn(
2022-11-23T03:10:42.0052219Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0053001Z   warnings.warn(
2022-11-23T03:10:42.0054300Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0055118Z   warnings.warn(
2022-11-23T03:10:42.0056197Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0057361Z   warnings.warn(
2022-11-23T03:10:42.0057624Z dist init r=2, world=4
2022-11-23T03:10:42.0057884Z dist init r=3, world=4
2022-11-23T03:10:42.0058118Z dist init r=0, world=4
2022-11-23T03:10:42.0058375Z dist init r=1, world=4
2022-11-23T03:10:42.0058620Z ok (4.818s)
2022-11-23T03:10:42.0058960Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0059685Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 33
2022-11-23T03:10:42.0060253Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11203
2022-11-23T03:10:42.0060694Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11204
2022-11-23T03:10:42.0061154Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11205
2022-11-23T03:10:42.0061602Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11206
2022-11-23T03:10:42.0062382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0062984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0063575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0064055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0064645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0065071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0065645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0066121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0066686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0067137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0067713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0068264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0068848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0069627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0070213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0070660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0071128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0071637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0072147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0072628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0073291Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0074215Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0074891Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0075536Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0076048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0076513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0077151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0077599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0078869Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0079644Z   warnings.warn(
2022-11-23T03:10:42.0080925Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0081676Z   warnings.warn(
2022-11-23T03:10:42.0082814Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0083738Z   warnings.warn(
2022-11-23T03:10:42.0084939Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0085713Z   warnings.warn(
2022-11-23T03:10:42.0085969Z dist init r=1, world=4
2022-11-23T03:10:42.0086201Z dist init r=3, world=4
2022-11-23T03:10:42.0086452Z dist init r=2, world=4
2022-11-23T03:10:42.0086704Z dist init r=0, world=4
2022-11-23T03:10:42.0086926Z ok (4.918s)
2022-11-23T03:10:42.0087282Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0088148Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 32
2022-11-23T03:10:42.0088665Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11504
2022-11-23T03:10:42.0089294Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11505
2022-11-23T03:10:42.0089756Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11506
2022-11-23T03:10:42.0090278Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11507
2022-11-23T03:10:42.0090879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0091341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0091926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0092532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0093110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0093544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0094105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0094545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0095106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0095537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0096261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0096905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0097520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0097975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0098530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0099155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0099607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0100098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0100565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0101049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0101688Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0102360Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0103088Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0103779Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0104528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0104993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0105471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0105937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0107217Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0108098Z   warnings.warn(
2022-11-23T03:10:42.0109233Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0110180Z   warnings.warn(
2022-11-23T03:10:42.0111467Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0112238Z   warnings.warn(
2022-11-23T03:10:42.0113407Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0114175Z   warnings.warn(
2022-11-23T03:10:42.0114411Z dist init r=3, world=4
2022-11-23T03:10:42.0114674Z dist init r=2, world=4
2022-11-23T03:10:42.0114927Z dist init r=0, world=4
2022-11-23T03:10:42.0115157Z dist init r=1, world=4
2022-11-23T03:10:42.0115403Z ok (4.918s)
2022-11-23T03:10:42.0115762Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0116455Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 31
2022-11-23T03:10:42.0117010Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11805
2022-11-23T03:10:42.0117473Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11806
2022-11-23T03:10:42.0117935Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11807
2022-11-23T03:10:42.0118372Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11808
2022-11-23T03:10:42.0118987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0119599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0120202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0120685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0121256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0121699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0122228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0122665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0123227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0123664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0124292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0124988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0125584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0126016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0126593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0127063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0127523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0128009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0128671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0129157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0129778Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0130458Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0131133Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0131803Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0132283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0132746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0133195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0133829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0135087Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0135867Z   warnings.warn(
2022-11-23T03:10:42.0137621Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0138448Z   warnings.warn(
2022-11-23T03:10:42.0139603Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0140533Z   warnings.warn(
2022-11-23T03:10:42.0141609Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0142422Z   warnings.warn(
2022-11-23T03:10:42.0142853Z dist init r=0, world=4
2022-11-23T03:10:42.0143117Z dist init r=1, world=4
2022-11-23T03:10:42.0143351Z dist init r=2, world=4
2022-11-23T03:10:42.0143603Z dist init r=3, world=4
2022-11-23T03:10:42.0143848Z ok (4.818s)
2022-11-23T03:10:42.0144185Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0144906Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 30
2022-11-23T03:10:42.0145475Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12106
2022-11-23T03:10:42.0145942Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12107
2022-11-23T03:10:42.0146388Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12108
2022-11-23T03:10:42.0146838Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12109
2022-11-23T03:10:42.0147601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0148019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0148567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0149176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0149754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0150210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0150810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0151287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0152005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0152442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0152997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0153456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0154000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0154437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0155049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0155525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0155948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0156436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0157099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0157579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0158251Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0158960Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0159657Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0160526Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0161041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0161508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0161973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0162824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0163862Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0163986Z   warnings.warn(
2022-11-23T03:10:42.0165138Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0165921Z   warnings.warn(
2022-11-23T03:10:42.0167065Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0167844Z   warnings.warn(
2022-11-23T03:10:42.0168986Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0169759Z   warnings.warn(
2022-11-23T03:10:42.0169990Z dist init r=1, world=4
2022-11-23T03:10:42.0170243Z dist init r=3, world=4
2022-11-23T03:10:42.0170562Z dist init r=0, world=4
2022-11-23T03:10:42.0170804Z dist init r=2, world=4
2022-11-23T03:10:42.0171043Z ok (4.818s)
2022-11-23T03:10:42.0171395Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0172092Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 29
2022-11-23T03:10:42.0172654Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12407
2022-11-23T03:10:42.0173115Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12408
2022-11-23T03:10:42.0173543Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12409
2022-11-23T03:10:42.0173995Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12410
2022-11-23T03:10:42.0174609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0175065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0175689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0176198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0176994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0177460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0178023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0178497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0179074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0179526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0180079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0180552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0181143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0181595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0182166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0182685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0183142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0183642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0184119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0184615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0185272Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0185946Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0186632Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0187316Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0187837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0188379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0188862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0189319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0190587Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0191381Z   warnings.warn(
2022-11-23T03:10:42.0192513Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0193376Z   warnings.warn(
2022-11-23T03:10:42.0194533Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0195291Z   warnings.warn(
2022-11-23T03:10:42.0196427Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0197185Z   warnings.warn(
2022-11-23T03:10:42.0197414Z dist init r=3, world=4
2022-11-23T03:10:42.0197665Z dist init r=2, world=4
2022-11-23T03:10:42.0197913Z dist init r=0, world=4
2022-11-23T03:10:42.0198140Z dist init r=1, world=4
2022-11-23T03:10:42.0198374Z ok (4.918s)
2022-11-23T03:10:42.0198725Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0199433Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 28
2022-11-23T03:10:42.0199975Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12708
2022-11-23T03:10:42.0200429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12709
2022-11-23T03:10:42.0200880Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12710
2022-11-23T03:10:42.0201305Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12711
2022-11-23T03:10:42.0201931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0202394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0202981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0203434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0204068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0204524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0205096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0205543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0206113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0206555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0207123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0207566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0208136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0208588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0209205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0209669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0210120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0210617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0211099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0211613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0212293Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0213004Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0213701Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0214385Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0214896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0215382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0215845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0216306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0217798Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0218578Z   warnings.warn(
2022-11-23T03:10:42.0219716Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0220483Z   warnings.warn(
2022-11-23T03:10:42.0221679Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0222471Z   warnings.warn(
2022-11-23T03:10:42.0223612Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0224422Z   warnings.warn(
2022-11-23T03:10:42.0224657Z dist init r=3, world=4
2022-11-23T03:10:42.0224906Z dist init r=0, world=4
2022-11-23T03:10:42.0225240Z dist init r=1, world=4
2022-11-23T03:10:42.0225484Z dist init r=2, world=4
2022-11-23T03:10:42.0225700Z ok (4.819s)
2022-11-23T03:10:42.0226046Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0226760Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 27
2022-11-23T03:10:42.0227293Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13009
2022-11-23T03:10:42.0227745Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13010
2022-11-23T03:10:42.0228195Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13011
2022-11-23T03:10:42.0228622Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13012
2022-11-23T03:10:42.0229241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0229694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0230264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0230875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0231435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0231860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0232406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0232839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0233399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0233999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0234551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0235009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0235582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0236017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0236560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0237173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0237839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0238349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0238829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0239323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0239979Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0240800Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0241464Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0242122Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0242629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0243132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0243747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0244212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0245482Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0246255Z   warnings.warn(
2022-11-23T03:10:42.0247379Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0248268Z   warnings.warn(
2022-11-23T03:10:42.0249541Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0250308Z   warnings.warn(
2022-11-23T03:10:42.0251450Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0252362Z   warnings.warn(
2022-11-23T03:10:42.0252576Z dist init r=3, world=4
2022-11-23T03:10:42.0252814Z dist init r=1, world=4
2022-11-23T03:10:42.0253050Z dist init r=0, world=4
2022-11-23T03:10:42.0253269Z dist init r=2, world=4
2022-11-23T03:10:42.0253494Z ok (4.918s)
2022-11-23T03:10:42.0253827Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0254541Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 26
2022-11-23T03:10:42.0255083Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13310
2022-11-23T03:10:42.0255517Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13311
2022-11-23T03:10:42.0255930Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13312
2022-11-23T03:10:42.0256364Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13313
2022-11-23T03:10:42.0257436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0257889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0258451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0258914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0259493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0260026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0260733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0261180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0261730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0262303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0262860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0263303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0263877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0264330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0264909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0265365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0265793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0266294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0266788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0267277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0267914Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0268607Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0269292Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0269968Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0270464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0270935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0271502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0271969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0273283Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0274071Z   warnings.warn(
2022-11-23T03:10:42.0275334Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0276062Z   warnings.warn(
2022-11-23T03:10:42.0277151Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0278127Z   warnings.warn(
2022-11-23T03:10:42.0279244Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0280008Z   warnings.warn(
2022-11-23T03:10:42.0280258Z dist init r=0, world=4
2022-11-23T03:10:42.0280504Z dist init r=1, world=4
2022-11-23T03:10:42.0280730Z dist init r=3, world=4
2022-11-23T03:10:42.0280971Z dist init r=2, world=4
2022-11-23T03:10:42.0281202Z ok (4.818s)
2022-11-23T03:10:42.0281529Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0282386Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 25
2022-11-23T03:10:42.0282970Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13611
2022-11-23T03:10:42.0283388Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13612
2022-11-23T03:10:42.0283821Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13613
2022-11-23T03:10:42.0284431Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13614
2022-11-23T03:10:42.0285046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0285487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0286054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0286525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0287076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0287502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0288055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0288576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0289302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0289899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0290463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0290909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0291468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0291912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0292477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0292932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0293396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0293954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0294452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0294926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0295586Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0296414Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0297473Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0298143Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0298668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0299138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0299587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0300064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0301676Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0302458Z   warnings.warn(
2022-11-23T03:10:42.0303591Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0304358Z   warnings.warn(
2022-11-23T03:10:42.0305575Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0306686Z   warnings.warn(
2022-11-23T03:10:42.0307830Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0308589Z   warnings.warn(
2022-11-23T03:10:42.0308818Z dist init r=0, world=4
2022-11-23T03:10:42.0309063Z dist init r=2, world=4
2022-11-23T03:10:42.0309305Z dist init r=3, world=4
2022-11-23T03:10:42.0309530Z dist init r=1, world=4
2022-11-23T03:10:42.0309761Z ok (4.918s)
2022-11-23T03:10:42.0310105Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0310799Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 24
2022-11-23T03:10:42.0311429Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13912
2022-11-23T03:10:42.0311879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13913
2022-11-23T03:10:42.0312309Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13914
2022-11-23T03:10:42.0312755Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13915
2022-11-23T03:10:42.0313358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0313802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0314348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0314793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0315362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0315832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0316402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0316864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0317434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0317857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0318420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0318880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0319451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0319873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0320432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0320887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0321318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0321963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0322437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0322965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0323592Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0324257Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0324969Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0325789Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0326286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0326754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0327216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0327683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0329165Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0329908Z   warnings.warn(
2022-11-23T03:10:42.0331000Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0331743Z   warnings.warn(
2022-11-23T03:10:42.0332833Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0333570Z   warnings.warn(
2022-11-23T03:10:42.0334835Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0335597Z   warnings.warn(
2022-11-23T03:10:42.0335842Z dist init r=0, world=4
2022-11-23T03:10:42.0336092Z dist init r=3, world=4
2022-11-23T03:10:42.0336317Z dist init r=1, world=4
2022-11-23T03:10:42.0336560Z dist init r=2, world=4
2022-11-23T03:10:42.0337007Z ok (4.818s)
2022-11-23T03:10:42.0337336Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0338047Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 23
2022-11-23T03:10:42.0338600Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14213
2022-11-23T03:10:42.0339034Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14214
2022-11-23T03:10:42.0339568Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14215
2022-11-23T03:10:42.0340021Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14216
2022-11-23T03:10:42.0340634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0341064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0341791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0342243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0342784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0343211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0343927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0344396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0345043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0345487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0346047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0346510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0347068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0347510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0348218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0348834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0349289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0349788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0350285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0350758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0351405Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0352088Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0352771Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0353435Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0353957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0354587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0355021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0355463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0356724Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0357472Z   warnings.warn(
2022-11-23T03:10:42.0358776Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0359543Z   warnings.warn(
2022-11-23T03:10:42.0360674Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0361497Z   warnings.warn(
2022-11-23T03:10:42.0362780Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0363692Z   warnings.warn(
2022-11-23T03:10:42.0363944Z dist init r=2, world=4
2022-11-23T03:10:42.0364171Z dist init r=3, world=4
2022-11-23T03:10:42.0364413Z dist init r=0, world=4
2022-11-23T03:10:42.0364655Z dist init r=1, world=4
2022-11-23T03:10:42.0364866Z ok (4.918s)
2022-11-23T03:10:42.0365217Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0365922Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 22
2022-11-23T03:10:42.0366450Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14514
2022-11-23T03:10:42.0366903Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14515
2022-11-23T03:10:42.0367352Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14516
2022-11-23T03:10:42.0367796Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14517
2022-11-23T03:10:42.0368385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0368828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0369555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0369996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0370747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0371184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0371750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0372199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0372772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0373211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0373827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0374302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0375025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0375446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0375976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0376418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0377233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0377740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0378220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0378709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0379469Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0380138Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0380828Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0381512Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0382029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0382479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0383005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0383465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0384729Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0385499Z   warnings.warn(
2022-11-23T03:10:42.0386625Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0387394Z   warnings.warn(
2022-11-23T03:10:42.0388675Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0389413Z   warnings.warn(
2022-11-23T03:10:42.0390778Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0391560Z   warnings.warn(
2022-11-23T03:10:42.0391786Z dist init r=3, world=4
2022-11-23T03:10:42.0392032Z dist init r=1, world=4
2022-11-23T03:10:42.0392278Z dist init r=2, world=4
2022-11-23T03:10:42.0392503Z dist init r=0, world=4
2022-11-23T03:10:42.0392737Z ok (4.818s)
2022-11-23T03:10:42.0393084Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0393774Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 21
2022-11-23T03:10:42.0394320Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14815
2022-11-23T03:10:42.0394927Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14816
2022-11-23T03:10:42.0395360Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14817
2022-11-23T03:10:42.0395837Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14818
2022-11-23T03:10:42.0396423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0396855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0397570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0398038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0398611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0399049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0399601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0400060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0400774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0401176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0401718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0402158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0402706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0403106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0403647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0404088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0404760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0405243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0405735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0406225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0406866Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0407553Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0408297Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0408995Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0409492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0409964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0410427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0411050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0412459Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0413291Z   warnings.warn(
2022-11-23T03:10:42.0414426Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0415198Z   warnings.warn(
2022-11-23T03:10:42.0416353Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0417332Z   warnings.warn(
2022-11-23T03:10:42.0418466Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0419218Z   warnings.warn(
2022-11-23T03:10:42.0419465Z dist init r=3, world=4
2022-11-23T03:10:42.0419713Z dist init r=2, world=4
2022-11-23T03:10:42.0419939Z dist init r=1, world=4
2022-11-23T03:10:42.0420189Z dist init r=0, world=4
2022-11-23T03:10:42.0420426Z ok (4.918s)
2022-11-23T03:10:42.0420757Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0421466Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 20
2022-11-23T03:10:42.0422169Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15116
2022-11-23T03:10:42.0422611Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15117
2022-11-23T03:10:42.0423025Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15118
2022-11-23T03:10:42.0423448Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15119
2022-11-23T03:10:42.0424039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0424588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0425341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0425814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0426391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0426817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0427385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0427841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0428395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0428839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0429561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0430078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0430616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0431037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0431568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0431984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0432397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0432863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0433325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0433777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0434571Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0435243Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0435908Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0436558Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0437065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0437683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0438118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0438735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0439980Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0440738Z   warnings.warn(
2022-11-23T03:10:42.0442054Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0442781Z   warnings.warn(
2022-11-23T03:10:42.0444064Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0444813Z   warnings.warn(
2022-11-23T03:10:42.0445927Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0446725Z   warnings.warn(
2022-11-23T03:10:42.0446952Z dist init r=3, world=4
2022-11-23T03:10:42.0447176Z dist init r=0, world=4
2022-11-23T03:10:42.0447404Z dist init r=2, world=4
2022-11-23T03:10:42.0447632Z dist init r=1, world=4
2022-11-23T03:10:42.0447839Z ok (4.818s)
2022-11-23T03:10:42.0448324Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0448987Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 19
2022-11-23T03:10:42.0449493Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15417
2022-11-23T03:10:42.0450095Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15418
2022-11-23T03:10:42.0450532Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15419
2022-11-23T03:10:42.0450968Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15420
2022-11-23T03:10:42.0451556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0452001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0452559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0453172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0453709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0454130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0454670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0455103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0455644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0456058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0456605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0457428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0458006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0458515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0458894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0459067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0459308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0459543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0459781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0460007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0460405Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0460962Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0461425Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0461796Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0462010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0462215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0462425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0462627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0463606Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0463709Z   warnings.warn(
2022-11-23T03:10:42.0464851Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0464957Z   warnings.warn(
2022-11-23T03:10:42.0465967Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0466071Z   warnings.warn(
2022-11-23T03:10:42.0467053Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0467152Z   warnings.warn(
2022-11-23T03:10:42.0467253Z dist init r=0, world=4
2022-11-23T03:10:42.0467403Z dist init r=2, world=4
2022-11-23T03:10:42.0467499Z dist init r=3, world=4
2022-11-23T03:10:42.0467599Z dist init r=1, world=4
2022-11-23T03:10:42.0467689Z ok (4.918s)
2022-11-23T03:10:42.0467903Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0468365Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 18
2022-11-23T03:10:42.0468579Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15718
2022-11-23T03:10:42.0468786Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15719
2022-11-23T03:10:42.0468984Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15720
2022-11-23T03:10:42.0469187Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15721
2022-11-23T03:10:42.0469552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0469724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0470296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0470643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0471002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0471170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0471537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0471710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0472063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0472228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0472594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0472770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0473130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0473296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0473655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0473822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0474060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0474300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0474693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0474912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0475294Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0475664Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0476034Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0476402Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0476666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0476875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0477088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0477292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0478257Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0478532Z   warnings.warn(
2022-11-23T03:10:42.0479517Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0479668Z   warnings.warn(
2022-11-23T03:10:42.0480671Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0480768Z   warnings.warn(
2022-11-23T03:10:42.0481903Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0482001Z   warnings.warn(
2022-11-23T03:10:42.0482097Z dist init r=3, world=4
2022-11-23T03:10:42.0482192Z dist init r=0, world=4
2022-11-23T03:10:42.0482278Z dist init r=2, world=4
2022-11-23T03:10:42.0482373Z dist init r=1, world=4
2022-11-23T03:10:42.0482460Z ok (5.019s)
2022-11-23T03:10:42.0482662Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0483267Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 17
2022-11-23T03:10:42.0483473Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16019
2022-11-23T03:10:42.0483677Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16020
2022-11-23T03:10:42.0483867Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16021
2022-11-23T03:10:42.0484066Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16022
2022-11-23T03:10:42.0484603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0484767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0485136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0485318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0485721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0485898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0486266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0486439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0486792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0486955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0487315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0487491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0487856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0488019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0488587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0488751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0488980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0489205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0489426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0489648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0490030Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0490572Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0490963Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0491350Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0491562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0491780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0491999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0492211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0493219Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0493487Z   warnings.warn(
2022-11-23T03:10:42.0494434Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0494529Z   warnings.warn(
2022-11-23T03:10:42.0495521Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0495627Z   warnings.warn(
2022-11-23T03:10:42.0496566Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0496659Z   warnings.warn(
2022-11-23T03:10:42.0496946Z dist init r=1, world=4
2022-11-23T03:10:42.0497118Z dist init r=0, world=4
2022-11-23T03:10:42.0497214Z dist init r=3, world=4
2022-11-23T03:10:42.0497309Z dist init r=2, world=4
2022-11-23T03:10:42.0497396Z ok (4.818s)
2022-11-23T03:10:42.0497602Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0498243Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 16
2022-11-23T03:10:42.0498452Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16320
2022-11-23T03:10:42.0498660Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16321
2022-11-23T03:10:42.0498855Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16322
2022-11-23T03:10:42.0499061Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16323
2022-11-23T03:10:42.0499424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0499594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0499962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0500142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0500497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0500818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0501165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0501492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0501867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0502049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0502421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0502598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0502949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0503110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0503475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0503644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0503944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0504191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0504426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0504649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0505043Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0505428Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0505810Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0506348Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0506552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0506822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0507028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0507405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0508413Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0508520Z   warnings.warn(
2022-11-23T03:10:42.0509498Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0509601Z   warnings.warn(
2022-11-23T03:10:42.0510575Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0510674Z   warnings.warn(
2022-11-23T03:10:42.0511795Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0511890Z   warnings.warn(
2022-11-23T03:10:42.0512162Z dist init r=1, world=4
2022-11-23T03:10:42.0512254Z dist init r=3, world=4
2022-11-23T03:10:42.0512352Z dist init r=0, world=4
2022-11-23T03:10:42.0512449Z dist init r=2, world=4
2022-11-23T03:10:42.0512538Z ok (4.819s)
2022-11-23T03:10:42.0512752Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0513265Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 15
2022-11-23T03:10:42.0513486Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16621
2022-11-23T03:10:42.0513694Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16622
2022-11-23T03:10:42.0513894Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16623
2022-11-23T03:10:42.0514103Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16624
2022-11-23T03:10:42.0514470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0514637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0515007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0515192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0515556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0515777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0516144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0516315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0516666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0516828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0517187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0517368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0517723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0517891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0518249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0518417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0518652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0518884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0519272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0519494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0519873Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0520252Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0520809Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0521191Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0521402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0521626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0521841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0522100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0523120Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0523386Z   warnings.warn(
2022-11-23T03:10:42.0524340Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0524480Z   warnings.warn(
2022-11-23T03:10:42.0525504Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0525601Z   warnings.warn(
2022-11-23T03:10:42.0526729Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0526831Z   warnings.warn(
2022-11-23T03:10:42.0526933Z dist init r=3, world=4
2022-11-23T03:10:42.0527023Z dist init r=0, world=4
2022-11-23T03:10:42.0527121Z dist init r=1, world=4
2022-11-23T03:10:42.0527217Z dist init r=2, world=4
2022-11-23T03:10:42.0527306Z ok (4.918s)
2022-11-23T03:10:42.0527516Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0527972Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 14
2022-11-23T03:10:42.0528183Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16922
2022-11-23T03:10:42.0528388Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16923
2022-11-23T03:10:42.0528589Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16924
2022-11-23T03:10:42.0528795Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16925
2022-11-23T03:10:42.0529167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0529331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0529857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0530034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0530374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0530531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0530871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0531098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0531451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0531608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0531954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0532124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0532469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0532624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0532969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0533135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0533359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0533678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0533902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0534120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0534500Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0535050Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0535435Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0535824Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0536041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0536271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0536509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0536957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0538028Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0538154Z   warnings.warn(
2022-11-23T03:10:42.0539146Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0539249Z   warnings.warn(
2022-11-23T03:10:42.0540309Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0540426Z   warnings.warn(
2022-11-23T03:10:42.0541405Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0541503Z   warnings.warn(
2022-11-23T03:10:42.0541605Z dist init r=3, world=4
2022-11-23T03:10:42.0541702Z dist init r=2, world=4
2022-11-23T03:10:42.0541799Z dist init r=0, world=4
2022-11-23T03:10:42.0541889Z dist init r=1, world=4
2022-11-23T03:10:42.0541979Z ok (4.818s)
2022-11-23T03:10:42.0542195Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0542655Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 13
2022-11-23T03:10:42.0542940Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17223
2022-11-23T03:10:42.0543147Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17224
2022-11-23T03:10:42.0543354Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17225
2022-11-23T03:10:42.0543548Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17226
2022-11-23T03:10:42.0543916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0544084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0544458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0544640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0545004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0545169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0545536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0545719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0546061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0546228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0546593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0546776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0547137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0547298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0547662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0547843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0548083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0548310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0548698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0548985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0549383Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0549757Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0550134Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0550689Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0550913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0551142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0551355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0551564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0552632Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0552742Z   warnings.warn(
2022-11-23T03:10:42.0553892Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0553999Z   warnings.warn(
2022-11-23T03:10:42.0554945Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0555045Z   warnings.warn(
2022-11-23T03:10:42.0555996Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0556100Z   warnings.warn(
2022-11-23T03:10:42.0556199Z dist init r=2, world=4
2022-11-23T03:10:42.0556293Z dist init r=1, world=4
2022-11-23T03:10:42.0556379Z dist init r=3, world=4
2022-11-23T03:10:42.0556474Z dist init r=0, world=4
2022-11-23T03:10:42.0556568Z ok (4.818s)
2022-11-23T03:10:42.0556777Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0557221Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 12
2022-11-23T03:10:42.0557425Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17524
2022-11-23T03:10:42.0557680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17525
2022-11-23T03:10:42.0557889Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17526
2022-11-23T03:10:42.0558081Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17527
2022-11-23T03:10:42.0558614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0558784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0559162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0559351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0559710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0559878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0560254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0560493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0560845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0561009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0561527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0561704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0562048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0562213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0562742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0562922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0563149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0563390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0563627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0563862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0564264Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0564656Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0565048Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0565436Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0565659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0565885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0566092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0566307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0567372Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0567486Z   warnings.warn(
2022-11-23T03:10:42.0568633Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0568732Z   warnings.warn(
2022-11-23T03:10:42.0569688Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0569841Z   warnings.warn(
2022-11-23T03:10:42.0570788Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0570885Z   warnings.warn(
2022-11-23T03:10:42.0571163Z dist init r=3, world=4
2022-11-23T03:10:42.0571263Z dist init r=0, world=4
2022-11-23T03:10:42.0571354Z dist init r=2, world=4
2022-11-23T03:10:42.0571455Z dist init r=1, world=4
2022-11-23T03:10:42.0571543Z ok (4.818s)
2022-11-23T03:10:42.0571757Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0572224Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 11
2022-11-23T03:10:42.0572438Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17825
2022-11-23T03:10:42.0572646Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17826
2022-11-23T03:10:42.0572854Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17827
2022-11-23T03:10:42.0573048Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17828
2022-11-23T03:10:42.0573412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0573582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0573955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0574144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0574501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0574667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0575188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0575368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0575698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0575857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0576265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0576449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0576997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0577160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0577516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0577694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0577911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0578142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0578374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0578592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0579242Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0579630Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0580022Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0580409Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0580627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0580854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0581063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0581275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0582424Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0582532Z   warnings.warn(
2022-11-23T03:10:42.0583566Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0583670Z   warnings.warn(
2022-11-23T03:10:42.0584620Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0584716Z   warnings.warn(
2022-11-23T03:10:42.0585924Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0586039Z   warnings.warn(
2022-11-23T03:10:42.0586142Z dist init r=1, world=4
2022-11-23T03:10:42.0586244Z dist init r=2, world=4
2022-11-23T03:10:42.0586333Z dist init r=3, world=4
2022-11-23T03:10:42.0586435Z dist init r=0, world=4
2022-11-23T03:10:42.0586527Z ok (4.918s)
2022-11-23T03:10:42.0586742Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0587208Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 10
2022-11-23T03:10:42.0587424Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18126
2022-11-23T03:10:42.0587633Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18127
2022-11-23T03:10:42.0587933Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18128
2022-11-23T03:10:42.0588129Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18129
2022-11-23T03:10:42.0588501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0588672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0589044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0589384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0589728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0589888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0590245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0590427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0590928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0591093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0591456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0591645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0592008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0592170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0592534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0592714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0592944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0593178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0593409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0593799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0594182Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0594609Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0594989Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0595361Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0595574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0595776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0595992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0596199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0597170Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0597332Z   warnings.warn(
2022-11-23T03:10:42.0598482Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0598588Z   warnings.warn(
2022-11-23T03:10:42.0599577Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0599680Z   warnings.warn(
2022-11-23T03:10:42.0600656Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0600760Z   warnings.warn(
2022-11-23T03:10:42.0600862Z dist init r=2, world=4
2022-11-23T03:10:42.0601124Z dist init r=3, world=4
2022-11-23T03:10:42.0601212Z dist init r=1, world=4
2022-11-23T03:10:42.0601307Z dist init r=0, world=4
2022-11-23T03:10:42.0601464Z ok (4.918s)
2022-11-23T03:10:42.0601675Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0602302Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 9
2022-11-23T03:10:42.0602513Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18427
2022-11-23T03:10:42.0602727Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18428
2022-11-23T03:10:42.0602932Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18429
2022-11-23T03:10:42.0603127Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18430
2022-11-23T03:10:42.0603554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0603742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0604138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0604334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0604700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0604878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0605255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0605451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0605792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0605971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0606347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0606753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0607117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0607291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0607834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0608026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0608253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0608508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0608758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0609003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0609405Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0609807Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0610210Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0610607Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0610844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0611053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0611292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0611516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0612539Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0612659Z   warnings.warn(
2022-11-23T03:10:42.0613711Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0613835Z   warnings.warn(
2022-11-23T03:10:42.0614832Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0614946Z   warnings.warn(
2022-11-23T03:10:42.0615932Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0616095Z   warnings.warn(
2022-11-23T03:10:42.0616210Z dist init r=2, world=4
2022-11-23T03:10:42.0616325Z dist init r=1, world=4
2022-11-23T03:10:42.0616415Z dist init r=0, world=4
2022-11-23T03:10:42.0616530Z dist init r=3, world=4
2022-11-23T03:10:42.0616632Z ok (4.919s)
2022-11-23T03:10:42.0617054Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0617537Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 8
2022-11-23T03:10:42.0617771Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18728
2022-11-23T03:10:42.0617994Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18729
2022-11-23T03:10:42.0618208Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18730
2022-11-23T03:10:42.0618402Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18731
2022-11-23T03:10:42.0618773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0618947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0619324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0619517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0619887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0620064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0620446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0620633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0620974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0621147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0621513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0621698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0622292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0622467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0622829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0623011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0623228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0623464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0623696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0623924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0624311Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0624739Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0625204Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0625583Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0625807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0626010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0626228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0626439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0627631Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0627752Z   warnings.warn(
2022-11-23T03:10:42.0628747Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0628857Z   warnings.warn(
2022-11-23T03:10:42.0629848Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0629963Z   warnings.warn(
2022-11-23T03:10:42.0631069Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0631224Z   warnings.warn(
2022-11-23T03:10:42.0631338Z dist init r=1, world=4
2022-11-23T03:10:42.0631445Z dist init r=3, world=4
2022-11-23T03:10:42.0631535Z dist init r=2, world=4
2022-11-23T03:10:42.0631640Z dist init r=0, world=4
2022-11-23T03:10:42.0631735Z ok (4.918s)
2022-11-23T03:10:42.0631949Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0632401Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 7
2022-11-23T03:10:42.0632612Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19029
2022-11-23T03:10:42.0632825Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19030
2022-11-23T03:10:42.0633036Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19031
2022-11-23T03:10:42.0633228Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19032
2022-11-23T03:10:42.0633593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0633819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0634189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0634376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0634732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0634902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0635262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0635601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0635970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0649260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0649716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0649906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0650276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0650441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0650812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0650992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0651231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0651473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0651713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0651939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0652340Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0652730Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0653115Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0653503Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0653836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0654076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0654288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0654500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0655522Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0655627Z   warnings.warn(
2022-11-23T03:10:42.0657135Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0657539Z   warnings.warn(
2022-11-23T03:10:42.0659651Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0659769Z   warnings.warn(
2022-11-23T03:10:42.0660780Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0660884Z   warnings.warn(
2022-11-23T03:10:42.0660989Z dist init r=2, world=4
2022-11-23T03:10:42.0661088Z dist init r=0, world=4
2022-11-23T03:10:42.0661179Z dist init r=3, world=4
2022-11-23T03:10:42.0661280Z dist init r=1, world=4
2022-11-23T03:10:42.0661369Z ok (4.818s)
2022-11-23T03:10:42.0661584Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0662049Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 6
2022-11-23T03:10:42.0662276Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19330
2022-11-23T03:10:42.0662486Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19331
2022-11-23T03:10:42.0662702Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19332
2022-11-23T03:10:42.0663052Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19333
2022-11-23T03:10:42.0663411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0663577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0664124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0664408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0664785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0664957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0665322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0665505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0665850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0666021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0666384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0666566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0666929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0667175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0667543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0667724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0667950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0668190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0668429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0668659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0669054Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0669600Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0669972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0670345Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0670564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0670771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0670981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0671189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0672375Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0672489Z   warnings.warn(
2022-11-23T03:10:42.0673535Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0673650Z   warnings.warn(
2022-11-23T03:10:42.0674637Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0674745Z   warnings.warn(
2022-11-23T03:10:42.0675862Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0675958Z   warnings.warn(
2022-11-23T03:10:42.0676114Z dist init r=0, world=4
2022-11-23T03:10:42.0676211Z dist init r=2, world=4
2022-11-23T03:10:42.0676299Z dist init r=3, world=4
2022-11-23T03:10:42.0676400Z dist init r=1, world=4
2022-11-23T03:10:42.0676490Z ok (4.818s)
2022-11-23T03:10:42.0676698Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0677142Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 5
2022-11-23T03:10:42.0677349Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19631
2022-11-23T03:10:42.0677552Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19632
2022-11-23T03:10:42.0677750Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19633
2022-11-23T03:10:42.0677942Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19634
2022-11-23T03:10:42.0678305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0678468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0678833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0679016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0679549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0679712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0680072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0680261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0680605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0680774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0681133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0681310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0681670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0681833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0682350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0682522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0682797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0683103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0683332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0683559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0683954Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0684331Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0684707Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0685083Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0685355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0685730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0685950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0686164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0687184Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0687291Z   warnings.warn(
2022-11-23T03:10:42.0688288Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0688393Z   warnings.warn(
2022-11-23T03:10:42.0689535Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0689634Z   warnings.warn(
2022-11-23T03:10:42.0690767Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0690869Z   warnings.warn(
2022-11-23T03:10:42.0690969Z dist init r=3, world=4
2022-11-23T03:10:42.0691075Z dist init r=1, world=4
2022-11-23T03:10:42.0691165Z dist init r=0, world=4
2022-11-23T03:10:42.0691271Z dist init r=2, world=4
2022-11-23T03:10:42.0691359Z ok (4.818s)
2022-11-23T03:10:42.0691660Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0692132Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 4
2022-11-23T03:10:42.0692355Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19932
2022-11-23T03:10:42.0692565Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19933
2022-11-23T03:10:42.0692773Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19934
2022-11-23T03:10:42.0692966Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19935
2022-11-23T03:10:42.0693335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0693503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0693872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0694217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0694621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0694780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0695136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0695316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0695647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0695805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0696154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0696333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0696673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0697116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0697489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0697664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0697884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0698109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0698333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0698741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0699140Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0699533Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0699917Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0700296Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0700520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0700730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0700952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0701245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0702406Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0702509Z   warnings.warn(
2022-11-23T03:10:42.0703460Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0703623Z   warnings.warn(
2022-11-23T03:10:42.0704824Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0704928Z   warnings.warn(
2022-11-23T03:10:42.0705918Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0706023Z   warnings.warn(
2022-11-23T03:10:42.0706129Z dist init r=1, world=4
2022-11-23T03:10:42.0706228Z dist init r=0, world=4
2022-11-23T03:10:42.0706318Z dist init r=3, world=4
2022-11-23T03:10:42.0706425Z dist init r=2, world=4
2022-11-23T03:10:42.0706516Z ok (4.818s)
2022-11-23T03:10:42.0706730Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0707188Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 3
2022-11-23T03:10:42.0707408Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20233
2022-11-23T03:10:42.0707622Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20234
2022-11-23T03:10:42.0707834Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20235
2022-11-23T03:10:42.0708031Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20236
2022-11-23T03:10:42.0708402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0708571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0708949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0709133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0709485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0709651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0710074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0710256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0710621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0710785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0711153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0711335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0711685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0711848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0712218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0712404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0712684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0712921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0713154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0713378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0713773Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0714159Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0714552Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0714940Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0715163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0715374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0715599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0715813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0716827Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0716938Z   warnings.warn(
2022-11-23T03:10:42.0717930Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0718030Z   warnings.warn(
2022-11-23T03:10:42.0719079Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0719195Z   warnings.warn(
2022-11-23T03:10:42.0720325Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0720426Z   warnings.warn(
2022-11-23T03:10:42.0720524Z dist init r=1, world=4
2022-11-23T03:10:42.0720622Z dist init r=2, world=4
2022-11-23T03:10:42.0720708Z dist init r=3, world=4
2022-11-23T03:10:42.0720803Z dist init r=0, world=4
2022-11-23T03:10:42.0720898Z ok (4.818s)
2022-11-23T03:10:42.0721108Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0721614Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 2
2022-11-23T03:10:42.0721817Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20534
2022-11-23T03:10:42.0722018Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20535
2022-11-23T03:10:42.0722217Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20536
2022-11-23T03:10:42.0722404Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20537
2022-11-23T03:10:42.0722761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0722923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0723292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0723471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0723817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0723986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0724341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0724525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0724900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0725055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0725406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0725579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0725933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0726089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0726435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0726605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0726824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0727228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0727517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0727754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0728152Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0728538Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0728923Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0729309Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0729535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0729746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0729972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0730255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0731392Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0731497Z   warnings.warn(
2022-11-23T03:10:42.0732450Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0732553Z   warnings.warn(
2022-11-23T03:10:42.0733514Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0733614Z   warnings.warn(
2022-11-23T03:10:42.0734570Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0734672Z   warnings.warn(
2022-11-23T03:10:42.0734941Z dist init r=3, world=4
2022-11-23T03:10:42.0735049Z dist init r=2, world=4
2022-11-23T03:10:42.0735140Z dist init r=1, world=4
2022-11-23T03:10:42.0735240Z dist init r=0, world=4
2022-11-23T03:10:42.0735333Z ok (4.918s)
2022-11-23T03:10:42.0735546Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-11-23T03:10:42.0736006Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 1
2022-11-23T03:10:42.0736223Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20835
2022-11-23T03:10:42.0736480Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20836
2022-11-23T03:10:42.0736905Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20837
2022-11-23T03:10:42.0737122Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20838
2022-11-23T03:10:42.0737502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0737666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0738040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0738225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0738580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0738898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0739435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0739699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0740059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0740220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0740586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0740766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0741122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:10:42.0741284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:10:42.0741650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:10:42.0741839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:10:42.0742222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:10:42.0742449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:10:42.0742678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:10:42.0742899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:10:42.0743449Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0743837Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0744227Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0744621Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:10:42.0744841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:10:42.0745053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:10:42.0745272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:10:42.0745481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:10:42.0746560Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0746677Z   warnings.warn(
2022-11-23T03:10:42.0747671Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0747771Z   warnings.warn(
2022-11-23T03:10:42.0748754Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0748906Z   warnings.warn(
2022-11-23T03:10:42.0750013Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T03:10:42.0750111Z   warnings.warn(
2022-11-23T03:10:42.0750209Z dist init r=3, world=4
2022-11-23T03:10:42.0750303Z dist init r=1, world=4
2022-11-23T03:10:42.0750394Z dist init r=2, world=4
2022-11-23T03:10:42.0750488Z dist init r=0, world=4
2022-11-23T03:10:42.0750572Z ok (4.818s)
2022-11-23T03:10:42.0750777Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=True) (__main__.TestPureFP16)
2022-11-23T03:10:42.0751401Z Tests pure FP16 training, including when the parameter's dtype is ...     test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) succeeded - num_retries_left: 0
2022-11-23T03:10:42.0751766Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:10:42.0751789Z 
2022-11-23T03:10:42.0752054Z ----------------------------------------------------------------------
2022-11-23T03:10:42.0752162Z Ran 52 tests in 250.181s
2022-11-23T03:10:42.0752182Z 
2022-11-23T03:10:42.0752271Z OK (skipped=1)
2022-11-23T03:10:42.0752289Z 
2022-11-23T03:10:42.0752402Z Generating XML reports...
2022-11-23T03:10:42.0752822Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_pure_fp16/TEST-TestPureFP16-20221123030631.xml
2022-11-23T03:10:42.0752842Z 
2022-11-23T03:10:42.0753262Z ##[endgroup]
2022-11-23T03:10:42.0753734Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_pure_fp16 (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_pure_fp16_vfceuz13)
2022-11-23T03:10:42.0753753Z 
2022-11-23T03:10:42.3102460Z 
2022-11-23T03:10:42.3102995Z real	4m18.131s
2022-11-23T03:10:42.3103234Z user	14m22.787s
2022-11-23T03:10:42.3103352Z sys	9m20.799s
2022-11-23T03:10:42.3103528Z + for f in test/distributed/fsdp/*.py
2022-11-23T03:10:42.3103996Z + python test/run_test.py --verbose -i distributed/fsdp/test_fsdp_sharded_grad_scaler.py
2022-11-23T03:10:44.6765845Z Ignoring disabled issues:  []
2022-11-23T03:10:44.7294010Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:10:44.7294604Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:10:44.7295265Z Selected tests:
2022-11-23T03:10:44.7295619Z  distributed/fsdp/test_fsdp_sharded_grad_scaler.py
2022-11-23T03:10:44.7320399Z Prioritized test from test file changes.
2022-11-23T03:10:44.7320722Z reordering tests for PR:
2022-11-23T03:10:44.7321144Z prioritized: []
2022-11-23T03:10:44.7321709Z the rest: ['distributed/fsdp/test_fsdp_sharded_grad_scaler.py']
2022-11-23T03:10:44.7321953Z 
2022-11-23T03:10:44.7322509Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:10:44.7323444Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:10:44.7329880Z parallel (file granularity) tests:
2022-11-23T03:10:44.7330193Z  
2022-11-23T03:10:44.7330466Z serial (file granularity) tests:
2022-11-23T03:10:44.7330805Z  distributed/fsdp/test_fsdp_sharded_grad_scaler.py
2022-11-23T03:10:47.0291567Z Ignoring disabled issues:  []
2022-11-23T03:10:47.0437367Z Ignoring disabled issues:  []
2022-11-23T03:10:47.4418544Z Running distributed/fsdp/test_fsdp_sharded_grad_scaler.py ... [2022-11-23 03:10:47.441102]
2022-11-23T03:10:47.4419726Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_sharded_grad_scaler.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:10:47.441599]
2022-11-23T03:10:51.6281578Z 
2022-11-23T03:10:51.6282328Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_sharded_grad_scaler
2022-11-23T03:10:51.6284665Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_sharded_grad_scaler (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_sharded_grad_scaler_yj5i7ihf)
2022-11-23T03:10:51.6285149Z 
2022-11-23T03:10:51.6285290Z Running tests...
2022-11-23T03:10:51.6285884Z ----------------------------------------------------------------------
2022-11-23T03:10:51.6286517Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler
2022-11-23T03:10:51.6287047Z   test_grad_scaling (__main__.TestShardGradScaler) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:10:51.6287708Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.723s)
2022-11-23T03:10:51.6288524Z   test_inf_gradients_skip_optim_step (__main__.TestShardGradScaler) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:10:51.6289412Z   test_scaling_unscaling_sparse (__main__.TestShardGradScaler) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:10:51.6290413Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:10:51.6291511Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_none (__main__.TestShardedGradScalerParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:10:51.6292761Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:10:51.6293827Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_none (__main__.TestShardedGradScalerParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:10:51.6295210Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:10:51.6296276Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_none (__main__.TestShardedGradScalerParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:10:51.6297645Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:10:51.6298703Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_none (__main__.TestShardedGradScalerParityWithDDP) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:10:51.6299338Z 
2022-11-23T03:10:51.6299617Z ----------------------------------------------------------------------
2022-11-23T03:10:51.6299957Z Ran 11 tests in 1.736s
2022-11-23T03:10:51.6300313Z 
2022-11-23T03:10:51.6300405Z OK (skipped=11)
2022-11-23T03:10:51.6300565Z 
2022-11-23T03:10:51.6300697Z Generating XML reports...
2022-11-23T03:10:51.6301344Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardGradScaler-20221123031049.xml
2022-11-23T03:10:51.6302251Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardedGradScalerParityWithDDP-20221123031049.xml
2022-11-23T03:10:51.6302678Z 
2022-11-23T03:10:51.6302985Z ##[endgroup]
2022-11-23T03:10:51.6303799Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_sharded_grad_scaler (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_sharded_grad_scaler_yj5i7ihf)
2022-11-23T03:10:51.6304174Z 
2022-11-23T03:10:51.9820812Z 
2022-11-23T03:10:51.9821686Z real	0m9.671s
2022-11-23T03:10:51.9822151Z user	0m16.110s
2022-11-23T03:10:51.9822341Z sys	0m12.220s
2022-11-23T03:10:51.9822586Z + for f in test/distributed/fsdp/*.py
2022-11-23T03:10:51.9823244Z + python test/run_test.py --verbose -i distributed/fsdp/test_fsdp_state_dict.py
2022-11-23T03:10:54.3286336Z Ignoring disabled issues:  []
2022-11-23T03:10:54.3809896Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:10:54.3810589Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:10:54.3811133Z Selected tests:
2022-11-23T03:10:54.3811458Z  distributed/fsdp/test_fsdp_state_dict.py
2022-11-23T03:10:54.3836559Z Prioritized test from test file changes.
2022-11-23T03:10:54.3836900Z reordering tests for PR:
2022-11-23T03:10:54.3837199Z prioritized: []
2022-11-23T03:10:54.3837751Z the rest: ['distributed/fsdp/test_fsdp_state_dict.py']
2022-11-23T03:10:54.3837956Z 
2022-11-23T03:10:54.3838401Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:10:54.3839389Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:10:54.3845867Z parallel (file granularity) tests:
2022-11-23T03:10:54.3846798Z  
2022-11-23T03:10:54.3847283Z serial (file granularity) tests:
2022-11-23T03:10:54.3847645Z  distributed/fsdp/test_fsdp_state_dict.py
2022-11-23T03:10:56.6402984Z Ignoring disabled issues:  []
2022-11-23T03:10:56.6797299Z Ignoring disabled issues:  []
2022-11-23T03:10:57.0774200Z Running distributed/fsdp/test_fsdp_state_dict.py ... [2022-11-23 03:10:57.076795]
2022-11-23T03:10:57.0777848Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_state_dict.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:10:57.077267]
2022-11-23T03:11:01.6216837Z 
2022-11-23T03:11:01.6217734Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_state_dict
2022-11-23T03:11:01.6219030Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_state_dict (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_state_dict_9ramzqvy)
2022-11-23T03:11:01.6219425Z 
2022-11-23T03:11:01.6219542Z Running tests...
2022-11-23T03:11:01.6220148Z ----------------------------------------------------------------------
2022-11-23T03:11:01.6220717Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict
2022-11-23T03:11:01.6221415Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6222071Z Tests that we can save a state_dict and load it into a blank model ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:11:01.6222772Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.766s)
2022-11-23T03:11:01.6223866Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6224963Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6225704Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6226454Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6227267Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6228087Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6228965Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6229923Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6230516Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6231376Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6232179Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6232939Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6233801Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6234735Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6235548Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6236433Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6237194Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6238173Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6239139Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6240033Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6240809Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6241660Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6242534Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6243365Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6244188Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6244918Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6245620Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6246451Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6247318Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6248186Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6249072Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6249865Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6250839Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6251626Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6252457Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6253332Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6254114Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6255090Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6255710Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6257215Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6257860Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6258855Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6259469Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6260333Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6261169Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6261943Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6262671Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6263729Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6264468Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6265328Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6266134Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6267188Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6267805Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6268666Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6269561Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6270495Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6271198Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6272169Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6272735Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6273586Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6274380Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6275122Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6275930Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6276845Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6277548Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6278413Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6279179Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6280057Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6280891Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6281722Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6282415Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6283451Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6284236Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6285089Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6285943Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6286778Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6287465Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6288298Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6288971Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6289804Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6290615Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6291326Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6292010Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6292828Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6293524Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6294403Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6295105Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6295946Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6296619Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6297644Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6298417Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6299421Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6300138Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6301015Z Tests that we can save a state_dict and load it into a blank model ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6301904Z   test_fsdp_state_dict_keys_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:01.6303069Z   test_fsdp_state_dict_keys_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:01.6304046Z   test_fsdp_state_dict_keys_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:01.6304861Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6305707Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6306418Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6307427Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6308088Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6308890Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6309599Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6310428Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6311093Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6311899Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6312532Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6313336Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6314073Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6314897Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6315574Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6316381Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6317050Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6317858Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6318512Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6319305Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6319943Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6320751Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6321410Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6322327Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6323033Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6323852Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6324608Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6325585Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6326409Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6327308Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6327870Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6328694Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6329582Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6330407Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6331162Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6332039Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6332712Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6333542Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6334188Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6335153Z Tests saving the state dict, zeroing a target model's parameters, and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6335985Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6337124Z Test that saving after some training results in params being updated as ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6337704Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6338526Z Test that saving after some training results in params being updated as ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6339201Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6340037Z Test that saving after some training results in params being updated as ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6340850Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6341720Z Test that saving after some training results in params being updated as ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6342452Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6343338Z Test that saving after some training results in params being updated as ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6343991Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6344802Z Test that saving after some training results in params being updated as ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6345511Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6346322Z Test that saving after some training results in params being updated as ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6347138Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6347975Z Test that saving after some training results in params being updated as ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6348644Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6349449Z Test that saving after some training results in params being updated as ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6350112Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6351077Z Test that saving after some training results in params being updated as ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6351713Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6352658Z Test that saving after some training results in params being updated as ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6353325Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6354145Z Test that saving after some training results in params being updated as ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6354805Z   test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6355748Z Tests that FSDP's state_dict can be loaded into a local model. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6356412Z   test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6357197Z Tests that FSDP's state_dict can be loaded into a local model. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6357831Z   test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6358784Z Tests that FSDP's state_dict can be loaded into a local model. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6359428Z   test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6360215Z Tests that FSDP's state_dict can be loaded into a local model. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6360854Z   test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6361848Z Tests that FSDP's state_dict can be loaded into a local model. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6362470Z   test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6363211Z Tests that FSDP's state_dict can be loaded into a local model. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6363900Z   test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6364681Z Tests that FSDP's state_dict can be loaded into a local model. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6365334Z   test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6366046Z Tests that FSDP's state_dict can be loaded into a local model. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6366601Z   test_state_dict_rank0_offload_save_load_flow_use_orig_params_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6367508Z Tests saving a model checkpoint only on rank 0 and loading it only ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6368092Z   test_state_dict_rank0_offload_save_load_flow_use_orig_params_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6368836Z Tests saving a model checkpoint only on rank 0 and loading it only ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6369730Z   test_state_dict_save_load_flow_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:01.6370815Z   test_state_dict_save_load_flow_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:01.6371710Z   test_state_dict_save_load_flow_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:01.6372697Z   test_state_dict_skip_module_state_dict_type_local_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:11:01.6373683Z   test_state_dict_skip_module_state_dict_type_sharded_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:11:01.6374631Z   test_state_dict_skip_module_state_dict_type_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:11:01.6375449Z   test_state_dict_type (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:01.6376397Z   test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_False (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6377915Z   test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_True (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:11:01.6378993Z   test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_False (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:11:01.6380055Z   test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_True (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6381229Z   test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_False (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:11:01.6382243Z   test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_True (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:11:01.6383432Z   test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_False (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6384475Z   test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_True (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:11:01.6385182Z   test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6386319Z Tests saving and loading a state dict for a model manually wrapped with ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6386945Z   test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6387750Z Tests saving and loading a state dict for a model manually wrapped with ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6388375Z   test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6389254Z Tests saving and loading a state dict for a model manually wrapped with ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6390024Z   test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2022-11-23T03:11:01.6390792Z Tests saving and loading a state dict for a model manually wrapped with ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:01.6391696Z   test_state_dict_with_shared_parameters_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:01.6392850Z   test_state_dict_with_shared_parameters_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:01.6393811Z   test_state_dict_with_shared_parameters_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:01.6394756Z   test_wrong_state_dict_config (__main__.TestFSDPStateDict) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:01.6395290Z 
2022-11-23T03:11:01.6395553Z ----------------------------------------------------------------------
2022-11-23T03:11:01.6395876Z Ran 116 tests in 2.010s
2022-11-23T03:11:01.6396036Z 
2022-11-23T03:11:01.6396142Z OK (skipped=116)
2022-11-23T03:11:01.6396296Z 
2022-11-23T03:11:01.6396399Z Generating XML reports...
2022-11-23T03:11:01.6396987Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict/TEST-TestFSDPStateDict-20221123031059.xml
2022-11-23T03:11:01.6397331Z 
2022-11-23T03:11:01.6397725Z ##[endgroup]
2022-11-23T03:11:01.6398304Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_state_dict (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_state_dict_9ramzqvy)
2022-11-23T03:11:01.6398825Z 
2022-11-23T03:11:02.0734002Z 
2022-11-23T03:11:02.0734480Z real	0m10.091s
2022-11-23T03:11:02.0734783Z user	0m17.587s
2022-11-23T03:11:02.0735056Z sys	0m15.069s
2022-11-23T03:11:02.0735331Z + for f in test/distributed/fsdp/*.py
2022-11-23T03:11:02.0735965Z + python test/run_test.py --verbose -i distributed/fsdp/test_fsdp_summon_full_params.py
2022-11-23T03:11:04.4589606Z Ignoring disabled issues:  []
2022-11-23T03:11:04.5116502Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:11:04.5117073Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:11:04.5117440Z Selected tests:
2022-11-23T03:11:04.5117777Z  distributed/fsdp/test_fsdp_summon_full_params.py
2022-11-23T03:11:04.5144687Z Prioritized test from test file changes.
2022-11-23T03:11:04.5145160Z reordering tests for PR:
2022-11-23T03:11:04.5145466Z prioritized: []
2022-11-23T03:11:04.5146026Z the rest: ['distributed/fsdp/test_fsdp_summon_full_params.py']
2022-11-23T03:11:04.5146135Z 
2022-11-23T03:11:04.5146779Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:11:04.5147633Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:11:04.5152825Z parallel (file granularity) tests:
2022-11-23T03:11:04.5153465Z  
2022-11-23T03:11:04.5153805Z serial (file granularity) tests:
2022-11-23T03:11:04.5154158Z  distributed/fsdp/test_fsdp_summon_full_params.py
2022-11-23T03:11:06.7748309Z Ignoring disabled issues:  []
2022-11-23T03:11:06.7761751Z Ignoring disabled issues:  []
2022-11-23T03:11:07.2933251Z Running distributed/fsdp/test_fsdp_summon_full_params.py ... [2022-11-23 03:11:07.292633]
2022-11-23T03:11:07.2934666Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_summon_full_params.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:11:07.293102]
2022-11-23T03:11:11.5254786Z 
2022-11-23T03:11:11.5255324Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_summon_full_params
2022-11-23T03:11:11.5256433Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_summon_full_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_summon_full_params_yqveok7l)
2022-11-23T03:11:11.5257399Z 
2022-11-23T03:11:11.5257517Z Running tests...
2022-11-23T03:11:11.5258070Z ----------------------------------------------------------------------
2022-11-23T03:11:11.5258676Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params
2022-11-23T03:11:11.5259534Z   test_cannot_summon_full_params_from_backward (__main__.TestSummonFullParams) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:11:11.5260240Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.706s)
2022-11-23T03:11:11.5260938Z   test_cannot_summon_full_params_from_forward (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:11.5261553Z   test_named_parameters_buffers_prefix__recurse_False (__main__.TestSummonFullParams)
2022-11-23T03:11:11.5262294Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5262855Z   test_named_parameters_buffers_prefix__recurse_True (__main__.TestSummonFullParams)
2022-11-23T03:11:11.5263602Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:11.5264194Z   test_named_parameters_buffers_prefix_test_prefix_recurse_False (__main__.TestSummonFullParams)
2022-11-23T03:11:11.5264949Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:11.5265512Z   test_named_parameters_buffers_prefix_test_prefix_recurse_True (__main__.TestSummonFullParams)
2022-11-23T03:11:11.5266265Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:11.5267237Z   test_params_are_unflattenned_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5268303Z   test_params_are_unflattenned_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5269333Z   test_params_are_unflattenned_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5270353Z   test_params_are_unflattenned_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5271510Z   test_params_are_unflattenned_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5272556Z   test_params_are_unflattenned_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5273633Z   test_params_are_unflattenned_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5274606Z   test_params_are_unflattenned_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5275758Z   test_params_count_and_value_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5276902Z   test_params_count_and_value_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5278462Z   test_params_count_and_value_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5279732Z   test_params_count_and_value_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5280643Z   test_params_count_and_value_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5281672Z   test_params_count_and_value_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5282695Z   test_params_count_and_value_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5283703Z   test_params_count_and_value_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5284327Z   test_raises_rank0_with_writeback (__main__.TestSummonFullParams)
2022-11-23T03:11:11.5285069Z Tests that ``summon_full_params()`` with both ``rank0_only=True`` ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:11.5286075Z   test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5287175Z   test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5288358Z   test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5289472Z   test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5290552Z   test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:11:11.5291742Z   test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5292808Z   test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5293905Z   test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5294832Z   test_summon_from_non_fsdp (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:11.5295800Z   test_summon_full_param_recursive_recurse_False_summon_outer_False_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5297034Z   test_summon_full_param_recursive_recurse_False_summon_outer_False_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5298068Z   test_summon_full_param_recursive_recurse_False_summon_outer_True_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5299096Z   test_summon_full_param_recursive_recurse_False_summon_outer_True_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5300130Z   test_summon_full_param_recursive_recurse_True_summon_outer_False_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5301160Z   test_summon_full_param_recursive_recurse_True_summon_outer_False_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5302155Z   test_summon_full_param_recursive_recurse_True_summon_outer_True_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5303189Z   test_summon_full_param_recursive_recurse_True_summon_outer_True_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5304256Z   test_summon_full_param_shard_value_mixed_precision_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5305202Z   test_summon_full_param_shard_value_mixed_precision_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:11.5306094Z   test_summon_full_param_writeback (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:11.5307040Z   test_summon_full_params_equivalence_rank0_only_False_offload_to_cpu_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:11.5308071Z   test_summon_full_params_equivalence_rank0_only_False_offload_to_cpu_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5309081Z   test_summon_full_params_equivalence_rank0_only_True_offload_to_cpu_False (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5310057Z   test_summon_full_params_equivalence_rank0_only_True_offload_to_cpu_True (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:11.5311015Z   test_summon_full_params_respects_reshard_after_forward (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:11.5311886Z   test_summon_single_param (__main__.TestSummonFullParams) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:11.5312396Z   test_with_grads_core (__main__.TestSummonFullParams)
2022-11-23T03:11:11.5313095Z Tests the core usage of ``summon_full_params(with_grads=True)``. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:11.5313626Z   test_with_grads_none_grads (__main__.TestSummonFullParams)
2022-11-23T03:11:11.5314339Z Tests that if all ranks' ``FlatParameter`` has ``None`` gradient, then ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:11.5315208Z   test_summon_full_param_writeback (__main__.TestSummonFullParamsNoShard) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:11.5315615Z 
2022-11-23T03:11:11.5315886Z ----------------------------------------------------------------------
2022-11-23T03:11:11.5316222Z Ran 52 tests in 1.796s
2022-11-23T03:11:11.5316386Z 
2022-11-23T03:11:11.5316497Z OK (skipped=52)
2022-11-23T03:11:11.5316639Z 
2022-11-23T03:11:11.5316766Z Generating XML reports...
2022-11-23T03:11:11.5317407Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParams-20221123031109.xml
2022-11-23T03:11:11.5318260Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParamsNoShard-20221123031109.xml
2022-11-23T03:11:11.5318656Z 
2022-11-23T03:11:11.5318956Z ##[endgroup]
2022-11-23T03:11:11.5319598Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_summon_full_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_summon_full_params_yqveok7l)
2022-11-23T03:11:11.5319985Z 
2022-11-23T03:11:11.8800674Z 
2022-11-23T03:11:11.8801193Z real	0m9.807s
2022-11-23T03:11:11.8801515Z user	0m16.764s
2022-11-23T03:11:11.8801748Z sys	0m14.159s
2022-11-23T03:11:11.8802421Z + for f in test/distributed/fsdp/*.py
2022-11-23T03:11:11.8803097Z + python test/run_test.py --verbose -i distributed/fsdp/test_fsdp_tp_integration.py
2022-11-23T03:11:14.2732666Z Ignoring disabled issues:  []
2022-11-23T03:11:14.3255328Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:11:14.3255913Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:11:14.3256263Z Selected tests:
2022-11-23T03:11:14.3256578Z  distributed/fsdp/test_fsdp_tp_integration.py
2022-11-23T03:11:14.3284739Z Prioritized test from test file changes.
2022-11-23T03:11:14.3285065Z reordering tests for PR:
2022-11-23T03:11:14.3285364Z prioritized: []
2022-11-23T03:11:14.3285902Z the rest: ['distributed/fsdp/test_fsdp_tp_integration.py']
2022-11-23T03:11:14.3286139Z 
2022-11-23T03:11:14.3286707Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:11:14.3287982Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:11:14.3293182Z parallel (file granularity) tests:
2022-11-23T03:11:14.3293483Z  
2022-11-23T03:11:14.3293738Z serial (file granularity) tests:
2022-11-23T03:11:14.3294061Z  distributed/fsdp/test_fsdp_tp_integration.py
2022-11-23T03:11:16.5887328Z Ignoring disabled issues:  []
2022-11-23T03:11:16.6282398Z Ignoring disabled issues:  []
2022-11-23T03:11:17.0500805Z Running distributed/fsdp/test_fsdp_tp_integration.py ... [2022-11-23 03:11:17.049369]
2022-11-23T03:11:17.0503274Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_tp_integration.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:11:17.049839]
2022-11-23T03:11:21.2879318Z 
2022-11-23T03:11:21.2879959Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_tp_integration
2022-11-23T03:11:21.2881000Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_tp_integration (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_tp_integration_uc_2fud_)
2022-11-23T03:11:21.2881395Z 
2022-11-23T03:11:21.2881512Z Running tests...
2022-11-23T03:11:21.2882054Z ----------------------------------------------------------------------
2022-11-23T03:11:21.2882651Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_tp_integration
2022-11-23T03:11:21.2883121Z   test_fsdp_tp_checkpoint_integration (__main__.TestTPFSDPIntegration)
2022-11-23T03:11:21.2883590Z Tests checkpointing for TP + FSDP integration. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:11:21.2884223Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.748s)
2022-11-23T03:11:21.2884840Z   test_fsdp_tp_integration_tensor_parallel_size_2_cpu_offload_CPUOffload(offload_params=False) (__main__.TestTPFSDPIntegration)
2022-11-23T03:11:21.2885639Z Tests training for TP + FSDP integration by comparing an FSDP-only ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:11:21.2886294Z   test_fsdp_tp_integration_tensor_parallel_size_2_cpu_offload_CPUOffload(offload_params=True) (__main__.TestTPFSDPIntegration)
2022-11-23T03:11:21.2887114Z Tests training for TP + FSDP integration by comparing an FSDP-only ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:21.2887768Z   test_fsdp_tp_integration_tensor_parallel_size_4_cpu_offload_CPUOffload(offload_params=False) (__main__.TestTPFSDPIntegration)
2022-11-23T03:11:21.2888878Z Tests training for TP + FSDP integration by comparing an FSDP-only ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:21.2889558Z   test_fsdp_tp_integration_tensor_parallel_size_4_cpu_offload_CPUOffload(offload_params=True) (__main__.TestTPFSDPIntegration)
2022-11-23T03:11:21.2890529Z Tests training for TP + FSDP integration by comparing an FSDP-only ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:21.2890897Z 
2022-11-23T03:11:21.2891163Z ----------------------------------------------------------------------
2022-11-23T03:11:21.2891471Z Ran 5 tests in 1.758s
2022-11-23T03:11:21.2891634Z 
2022-11-23T03:11:21.2891743Z OK (skipped=5)
2022-11-23T03:11:21.2891897Z 
2022-11-23T03:11:21.2892022Z Generating XML reports...
2022-11-23T03:11:21.2892642Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_tp_integration/TEST-TestTPFSDPIntegration-20221123031119.xml
2022-11-23T03:11:21.2893013Z 
2022-11-23T03:11:21.2893297Z ##[endgroup]
2022-11-23T03:11:21.2893919Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_tp_integration (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_tp_integration_uc_2fud_)
2022-11-23T03:11:21.2894404Z 
2022-11-23T03:11:21.6469865Z 
2022-11-23T03:11:21.6470333Z real	0m9.767s
2022-11-23T03:11:21.6470686Z user	0m16.320s
2022-11-23T03:11:21.6471160Z sys	0m13.773s
2022-11-23T03:11:21.6471447Z + for f in test/distributed/fsdp/*.py
2022-11-23T03:11:21.6472073Z + python test/run_test.py --verbose -i distributed/fsdp/test_fsdp_traversal.py
2022-11-23T03:11:24.0344264Z Ignoring disabled issues:  []
2022-11-23T03:11:24.0881319Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:11:24.0881893Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:11:24.0882324Z Selected tests:
2022-11-23T03:11:24.0882628Z  distributed/fsdp/test_fsdp_traversal.py
2022-11-23T03:11:24.0911504Z Prioritized test from test file changes.
2022-11-23T03:11:24.0911836Z reordering tests for PR:
2022-11-23T03:11:24.0912145Z prioritized: []
2022-11-23T03:11:24.0912644Z the rest: ['distributed/fsdp/test_fsdp_traversal.py']
2022-11-23T03:11:24.0912868Z 
2022-11-23T03:11:24.0913421Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:11:24.0914367Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:11:24.0920173Z parallel (file granularity) tests:
2022-11-23T03:11:24.0920450Z  
2022-11-23T03:11:24.0920702Z serial (file granularity) tests:
2022-11-23T03:11:24.0921029Z  distributed/fsdp/test_fsdp_traversal.py
2022-11-23T03:11:26.3143594Z Ignoring disabled issues:  []
2022-11-23T03:11:26.3445673Z Ignoring disabled issues:  []
2022-11-23T03:11:26.7603226Z Running distributed/fsdp/test_fsdp_traversal.py ... [2022-11-23 03:11:26.759721]
2022-11-23T03:11:26.7604077Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_traversal.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:11:26.760170]
2022-11-23T03:11:30.9751484Z 
2022-11-23T03:11:30.9752540Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_traversal
2022-11-23T03:11:30.9753588Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_traversal (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_traversal_3bq2zzzn)
2022-11-23T03:11:30.9753976Z 
2022-11-23T03:11:30.9754096Z Running tests...
2022-11-23T03:11:30.9754762Z ----------------------------------------------------------------------
2022-11-23T03:11:30.9755256Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal
2022-11-23T03:11:30.9756082Z   test_fsdp_modules (__main__.TestTraversal) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:11:30.9756765Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.762s)
2022-11-23T03:11:30.9757100Z 
2022-11-23T03:11:30.9757370Z ----------------------------------------------------------------------
2022-11-23T03:11:30.9757703Z Ran 1 test in 1.762s
2022-11-23T03:11:30.9757874Z 
2022-11-23T03:11:30.9757991Z OK (skipped=1)
2022-11-23T03:11:30.9758153Z 
2022-11-23T03:11:30.9758284Z Generating XML reports...
2022-11-23T03:11:30.9758895Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal/TEST-TestTraversal-20221123031128.xml
2022-11-23T03:11:30.9759221Z 
2022-11-23T03:11:30.9759547Z ##[endgroup]
2022-11-23T03:11:30.9760179Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_traversal (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_traversal_3bq2zzzn)
2022-11-23T03:11:30.9760555Z 
2022-11-23T03:11:31.3326585Z 
2022-11-23T03:11:31.3327089Z real	0m9.686s
2022-11-23T03:11:31.3327515Z user	0m16.936s
2022-11-23T03:11:31.3327764Z sys	0m15.314s
2022-11-23T03:11:31.3328477Z + for f in test/distributed/fsdp/*.py
2022-11-23T03:11:31.3329120Z + python test/run_test.py --verbose -i distributed/fsdp/test_fsdp_uneven.py
2022-11-23T03:11:33.7449533Z Ignoring disabled issues:  []
2022-11-23T03:11:33.7990318Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:11:33.7991327Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:11:33.7991874Z Selected tests:
2022-11-23T03:11:33.7992168Z  distributed/fsdp/test_fsdp_uneven.py
2022-11-23T03:11:33.8017147Z Prioritized test from test file changes.
2022-11-23T03:11:33.8017848Z reordering tests for PR:
2022-11-23T03:11:33.8018126Z prioritized: []
2022-11-23T03:11:33.8018619Z the rest: ['distributed/fsdp/test_fsdp_uneven.py']
2022-11-23T03:11:33.8018817Z 
2022-11-23T03:11:33.8019382Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:11:33.8020340Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:11:33.8026787Z parallel (file granularity) tests:
2022-11-23T03:11:33.8027089Z  
2022-11-23T03:11:33.8027335Z serial (file granularity) tests:
2022-11-23T03:11:33.8027650Z  distributed/fsdp/test_fsdp_uneven.py
2022-11-23T03:11:36.0394813Z Ignoring disabled issues:  []
2022-11-23T03:11:36.0653801Z Ignoring disabled issues:  []
2022-11-23T03:11:36.4347818Z Running distributed/fsdp/test_fsdp_uneven.py ... [2022-11-23 03:11:36.434151]
2022-11-23T03:11:36.4349208Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_uneven.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:11:36.434612]
2022-11-23T03:11:40.6090588Z 
2022-11-23T03:11:40.6091220Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_uneven
2022-11-23T03:11:40.6092556Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_uneven (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_uneven_vcn7p2u3)
2022-11-23T03:11:40.6092928Z 
2022-11-23T03:11:40.6093045Z Running tests...
2022-11-23T03:11:40.6093590Z ----------------------------------------------------------------------
2022-11-23T03:11:40.6094154Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven
2022-11-23T03:11:40.6094609Z   test_one_iteration (__main__.TestUnevenParamShard)
2022-11-23T03:11:40.6095027Z Test FSDP with uneven divide of parameter shards. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:11:40.6095677Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.759s)
2022-11-23T03:11:40.6096311Z 
2022-11-23T03:11:40.6096631Z ----------------------------------------------------------------------
2022-11-23T03:11:40.6097408Z Ran 1 test in 1.759s
2022-11-23T03:11:40.6097556Z 
2022-11-23T03:11:40.6097677Z OK (skipped=1)
2022-11-23T03:11:40.6097825Z 
2022-11-23T03:11:40.6097953Z Generating XML reports...
2022-11-23T03:11:40.6098830Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven/TEST-TestUnevenParamShard-20221123031138.xml
2022-11-23T03:11:40.6099444Z 
2022-11-23T03:11:40.6100044Z ##[endgroup]
2022-11-23T03:11:40.6101176Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_uneven (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_uneven_vcn7p2u3)
2022-11-23T03:11:40.6101769Z 
2022-11-23T03:11:40.9699230Z 
2022-11-23T03:11:40.9699716Z real	0m9.637s
2022-11-23T03:11:40.9700019Z user	0m17.957s
2022-11-23T03:11:40.9700252Z sys	0m15.813s
2022-11-23T03:11:40.9700552Z + for f in test/distributed/fsdp/*.py
2022-11-23T03:11:40.9701231Z + python test/run_test.py --verbose -i distributed/fsdp/test_fsdp_use_orig_params.py
2022-11-23T03:11:43.3842398Z Ignoring disabled issues:  []
2022-11-23T03:11:43.4373344Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:11:43.4373941Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:11:43.4374281Z Selected tests:
2022-11-23T03:11:43.4374737Z  distributed/fsdp/test_fsdp_use_orig_params.py
2022-11-23T03:11:43.4401053Z Prioritized test from test file changes.
2022-11-23T03:11:43.4401642Z reordering tests for PR:
2022-11-23T03:11:43.4401945Z prioritized: []
2022-11-23T03:11:43.4402561Z the rest: ['distributed/fsdp/test_fsdp_use_orig_params.py']
2022-11-23T03:11:43.4402742Z 
2022-11-23T03:11:43.4403302Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:11:43.4404237Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:11:43.4410402Z parallel (file granularity) tests:
2022-11-23T03:11:43.4410695Z  
2022-11-23T03:11:43.4410959Z serial (file granularity) tests:
2022-11-23T03:11:43.4411283Z  distributed/fsdp/test_fsdp_use_orig_params.py
2022-11-23T03:11:45.7455592Z Ignoring disabled issues:  []
2022-11-23T03:11:45.7823902Z Ignoring disabled issues:  []
2022-11-23T03:11:46.1612011Z Running distributed/fsdp/test_fsdp_use_orig_params.py ... [2022-11-23 03:11:46.160538]
2022-11-23T03:11:46.1613260Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_use_orig_params.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:11:46.161000]
2022-11-23T03:11:50.4240550Z 
2022-11-23T03:11:50.4241160Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_use_orig_params
2022-11-23T03:11:50.4242211Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_use_orig_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_use_orig_params_xkv5agkb)
2022-11-23T03:11:50.4242666Z 
2022-11-23T03:11:50.4242787Z Running tests...
2022-11-23T03:11:50.4243358Z ----------------------------------------------------------------------
2022-11-23T03:11:50.4243929Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params
2022-11-23T03:11:50.4244520Z   test_named_parameters_in_forward (__main__.TestFSDPUseOrigParamsFQNs)
2022-11-23T03:11:50.4245029Z Tests that calling ``named_parameters()`` during forward returns FQNs ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:11:50.4245705Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.773s)
2022-11-23T03:11:50.4246521Z   test_param_and_buffer_names (__main__.TestFSDPUseOrigParamsFQNs)
2022-11-23T03:11:50.4247227Z Tests that, for ``use_orig_params=True``, the parameter and buffer ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:50.4247920Z   test_diff_hyperparams_cpu_offload_sharding_strategy_str_full_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2022-11-23T03:11:50.4248767Z Tests FSDP parity with DDP when using multiple parameter groups with ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:50.4249428Z   test_diff_hyperparams_cpu_offload_sharding_strategy_str_no_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2022-11-23T03:11:50.4250262Z Tests FSDP parity with DDP when using multiple parameter groups with ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:50.4250949Z   test_diff_hyperparams_cpu_offload_sharding_strategy_str_shard_grad_op (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2022-11-23T03:11:50.4251921Z Tests FSDP parity with DDP when using multiple parameter groups with ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:50.4252560Z   test_diff_hyperparams_sharding_strategy_str_full_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2022-11-23T03:11:50.4253393Z Tests FSDP parity with DDP when using multiple parameter groups with ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:50.4254060Z   test_diff_hyperparams_sharding_strategy_str_no_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2022-11-23T03:11:50.4254885Z Tests FSDP parity with DDP when using multiple parameter groups with ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:50.4255525Z   test_diff_hyperparams_sharding_strategy_str_shard_grad_op (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2022-11-23T03:11:50.4256362Z Tests FSDP parity with DDP when using multiple parameter groups with ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:50.4257499Z   test_diff_trainability (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2022-11-23T03:11:50.4258313Z Tests FSDP parity with DDP when using multiple parameter groups and ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:50.4258909Z   test_multiple_optimizers (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2022-11-23T03:11:50.4259697Z Tests using two optimizers where only one sets gradients to ``None``. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:50.4260291Z   test_access_params_after_forward (__main__.TestFSDPUseOrigParamsParamAccess)
2022-11-23T03:11:50.4260963Z Tests that accessing the original parameters after the forward but ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:50.4261589Z   test_multiple_forward_offload_params_False (__main__.TestFSDPUseOrigParamsUnshardReshard)
2022-11-23T03:11:50.4262347Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:50.4262950Z   test_multiple_forward_offload_params_True (__main__.TestFSDPUseOrigParamsUnshardReshard)
2022-11-23T03:11:50.4263723Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:50.4264437Z   test_summon_between_two_forwards_offload_params_False (__main__.TestFSDPUseOrigParamsUnshardReshard)
2022-11-23T03:11:50.4265220Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:50.4265845Z   test_summon_between_two_forwards_offload_params_True (__main__.TestFSDPUseOrigParamsUnshardReshard)
2022-11-23T03:11:50.4266633Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:50.4267193Z   test_grad_writeback (__main__.TestFSDPUseOrigParamsWriteback)
2022-11-23T03:11:50.4267912Z Tests that changes to the original parameters' gradients are written ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:50.4268496Z   test_param_writeback (__main__.TestFSDPUseOrigParamsWriteback)
2022-11-23T03:11:50.4269233Z Tests that changes to the original parameters are written back. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:11:50.4270229Z   test_writeback_shape_mismatch (__main__.TestFSDPUseOrigParamsWriteback) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:11:50.4270647Z 
2022-11-23T03:11:50.4270897Z ----------------------------------------------------------------------
2022-11-23T03:11:50.4271239Z Ran 18 tests in 1.790s
2022-11-23T03:11:50.4271407Z 
2022-11-23T03:11:50.4271520Z OK (skipped=18)
2022-11-23T03:11:50.4271682Z 
2022-11-23T03:11:50.4271813Z Generating XML reports...
2022-11-23T03:11:50.4272449Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsFQNs-20221123031148.xml
2022-11-23T03:11:50.4273389Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsMultipleParamGroups-20221123031148.xml
2022-11-23T03:11:50.4274339Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsParamAccess-20221123031148.xml
2022-11-23T03:11:50.4275269Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsUnshardReshard-20221123031148.xml
2022-11-23T03:11:50.4276219Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsWriteback-20221123031148.xml
2022-11-23T03:11:50.4276623Z 
2022-11-23T03:11:50.4276951Z ##[endgroup]
2022-11-23T03:11:50.4277602Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_use_orig_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_use_orig_params_xkv5agkb)
2022-11-23T03:11:50.4277983Z 
2022-11-23T03:11:50.7902102Z 
2022-11-23T03:11:50.7902928Z real	0m9.820s
2022-11-23T03:11:50.7903603Z user	0m18.821s
2022-11-23T03:11:50.7903887Z sys	0m16.375s
2022-11-23T03:11:50.7904219Z + for f in test/distributed/fsdp/*.py
2022-11-23T03:11:50.7904841Z + python test/run_test.py --verbose -i distributed/fsdp/test_shard_utils.py
2022-11-23T03:11:53.1712528Z Ignoring disabled issues:  []
2022-11-23T03:11:53.2288637Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:11:53.2289283Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:11:53.2289670Z Selected tests:
2022-11-23T03:11:53.2289945Z  distributed/fsdp/test_shard_utils.py
2022-11-23T03:11:53.2315714Z Prioritized test from test file changes.
2022-11-23T03:11:53.2316076Z reordering tests for PR:
2022-11-23T03:11:53.2316394Z prioritized: []
2022-11-23T03:11:53.2316904Z the rest: ['distributed/fsdp/test_shard_utils.py']
2022-11-23T03:11:53.2317138Z 
2022-11-23T03:11:53.2317947Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:11:53.2318964Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:11:53.2325655Z parallel (file granularity) tests:
2022-11-23T03:11:53.2325953Z  
2022-11-23T03:11:53.2326223Z serial (file granularity) tests:
2022-11-23T03:11:53.2326547Z  distributed/fsdp/test_shard_utils.py
2022-11-23T03:11:55.5613429Z Ignoring disabled issues:  []
2022-11-23T03:11:55.5765126Z Ignoring disabled issues:  []
2022-11-23T03:11:56.0489257Z Running distributed/fsdp/test_shard_utils.py ... [2022-11-23 03:11:56.048271]
2022-11-23T03:11:56.0490451Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_shard_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:11:56.048729]
2022-11-23T03:11:58.3615118Z 
2022-11-23T03:11:58.3616264Z Expand the folded group to see the log file of distributed/fsdp/test_shard_utils
2022-11-23T03:11:58.3617808Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_shard_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_shard_utils_hgo8sst0)
2022-11-23T03:11:58.3618180Z 
2022-11-23T03:11:58.3618394Z ##[endgroup]
2022-11-23T03:11:58.3619180Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_shard_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_shard_utils_hgo8sst0)
2022-11-23T03:11:58.3619542Z 
2022-11-23T03:11:58.7028082Z 
2022-11-23T03:11:58.7028740Z real	0m7.912s
2022-11-23T03:11:58.7029066Z user	0m16.541s
2022-11-23T03:11:58.7029320Z sys	0m14.301s
2022-11-23T03:11:58.7029620Z + for f in test/distributed/fsdp/*.py
2022-11-23T03:11:58.7030170Z + python test/run_test.py --verbose -i distributed/fsdp/test_utils.py
2022-11-23T03:12:01.1276660Z Ignoring disabled issues:  []
2022-11-23T03:12:01.1819907Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:12:01.1820520Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:12:01.1820892Z Selected tests:
2022-11-23T03:12:01.1821156Z  distributed/fsdp/test_utils.py
2022-11-23T03:12:01.1847155Z Prioritized test from test file changes.
2022-11-23T03:12:01.1847519Z reordering tests for PR:
2022-11-23T03:12:01.1847820Z prioritized: []
2022-11-23T03:12:01.1848319Z the rest: ['distributed/fsdp/test_utils.py']
2022-11-23T03:12:01.1848535Z 
2022-11-23T03:12:01.1849090Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:12:01.1850065Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:12:01.1855697Z parallel (file granularity) tests:
2022-11-23T03:12:01.1856001Z  
2022-11-23T03:12:01.1856268Z serial (file granularity) tests:
2022-11-23T03:12:01.1856674Z  distributed/fsdp/test_utils.py
2022-11-23T03:12:03.5448495Z Ignoring disabled issues:  []
2022-11-23T03:12:03.5553717Z Ignoring disabled issues:  []
2022-11-23T03:12:03.9641662Z Running distributed/fsdp/test_utils.py ... [2022-11-23 03:12:03.963540]
2022-11-23T03:12:03.9642949Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:12:03.963993]
2022-11-23T03:12:08.1858619Z 
2022-11-23T03:12:08.1859566Z Expand the folded group to see the log file of distributed/fsdp/test_utils
2022-11-23T03:12:08.1861281Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_utils_6m4fd406)
2022-11-23T03:12:08.1862235Z 
2022-11-23T03:12:08.1862482Z Running tests...
2022-11-23T03:12:08.1863421Z ----------------------------------------------------------------------
2022-11-23T03:12:08.1864484Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_utils
2022-11-23T03:12:08.1865651Z   test_module_wrap_policy (__main__.TestGetSubmoduleToStates)
2022-11-23T03:12:08.1867090Z Tests the module wrap policy on a nested model with buffers and a ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.763s)
2022-11-23T03:12:08.1868190Z   test_apply_to_tensors_cpu_cuda (__main__.TestUtils) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:12:08.1869036Z   test_apply_to_tensors_devices_['cpu'] (__main__.TestUtils) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:12:08.1869883Z   test_apply_to_tensors_devices_['cuda'] (__main__.TestUtils) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:12:08.1870545Z   test_packed_sequence (__main__.TestUtils)
2022-11-23T03:12:08.1871207Z Test to ensure RNN packed sequences are modified correctly. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:08.1872113Z   test_replace_by_prefix (__main__.TestUtils) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:08.1872455Z 
2022-11-23T03:12:08.1872649Z ----------------------------------------------------------------------
2022-11-23T03:12:08.1872989Z Ran 6 tests in 1.772s
2022-11-23T03:12:08.1873158Z 
2022-11-23T03:12:08.1873247Z OK (skipped=6)
2022-11-23T03:12:08.1873407Z 
2022-11-23T03:12:08.1873534Z Generating XML reports...
2022-11-23T03:12:08.1874157Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestGetSubmoduleToStates-20221123031205.xml
2022-11-23T03:12:08.1874906Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestUtils-20221123031205.xml
2022-11-23T03:12:08.1875210Z 
2022-11-23T03:12:08.1875545Z ##[endgroup]
2022-11-23T03:12:08.1876136Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_utils_6m4fd406)
2022-11-23T03:12:08.1876479Z 
2022-11-23T03:12:08.5396048Z 
2022-11-23T03:12:08.5396502Z real	0m9.837s
2022-11-23T03:12:08.5396803Z user	0m18.751s
2022-11-23T03:12:08.5397076Z sys	0m16.390s
2022-11-23T03:12:08.5397348Z + for f in test/distributed/fsdp/*.py
2022-11-23T03:12:08.5397842Z + python test/run_test.py --verbose -i distributed/fsdp/test_wrap.py
2022-11-23T03:12:10.9042728Z Ignoring disabled issues:  []
2022-11-23T03:12:10.9585192Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:12:10.9585808Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:12:10.9586189Z Selected tests:
2022-11-23T03:12:10.9586446Z  distributed/fsdp/test_wrap.py
2022-11-23T03:12:10.9609448Z Prioritized test from test file changes.
2022-11-23T03:12:10.9609876Z reordering tests for PR:
2022-11-23T03:12:10.9610194Z prioritized: []
2022-11-23T03:12:10.9610675Z the rest: ['distributed/fsdp/test_wrap.py']
2022-11-23T03:12:10.9610874Z 
2022-11-23T03:12:10.9611339Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:12:10.9612297Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:12:10.9618225Z parallel (file granularity) tests:
2022-11-23T03:12:10.9618559Z  
2022-11-23T03:12:10.9618806Z serial (file granularity) tests:
2022-11-23T03:12:10.9619124Z  distributed/fsdp/test_wrap.py
2022-11-23T03:12:13.2707321Z Ignoring disabled issues:  []
2022-11-23T03:12:13.2716465Z Ignoring disabled issues:  []
2022-11-23T03:12:13.6553414Z Running distributed/fsdp/test_wrap.py ... [2022-11-23 03:12:13.654694]
2022-11-23T03:12:13.6554209Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_wrap.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:12:13.655166]
2022-11-23T03:12:17.8923935Z 
2022-11-23T03:12:17.8924700Z Expand the folded group to see the log file of distributed/fsdp/test_wrap
2022-11-23T03:12:17.8925710Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_wrap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_wrap_o2uqfxjr)
2022-11-23T03:12:17.8926057Z 
2022-11-23T03:12:17.8926151Z Running tests...
2022-11-23T03:12:17.8926723Z ----------------------------------------------------------------------
2022-11-23T03:12:17.8927275Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_wrap
2022-11-23T03:12:17.8927991Z   test_always_wrap (__main__.TestAutoWrap)
2022-11-23T03:12:17.8928387Z Test to ensure that if `always_wrap_policy` is ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:12:17.8929020Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.671s)
2022-11-23T03:12:17.8929894Z   test_always_wrap_with_ignored_modules_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:12:17.8930854Z   test_always_wrap_with_ignored_modules_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:12:17.8931386Z   test_auto_wrap_api (__main__.TestAutoWrap)
2022-11-23T03:12:17.8932114Z Test to ensure with auto wrap, we wrap child modules correctly based on the min_num_params. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:17.8932746Z   test_auto_wrap_preset_exclude_wrap (__main__.TestAutoWrap)
2022-11-23T03:12:17.8933531Z Test to ensure excluded modules are not wrapped, regardless if the total param size is greater than the ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:17.8934114Z   test_auto_wrap_preset_exclude_wrap_include_children (__main__.TestAutoWrap)
2022-11-23T03:12:17.8934900Z Test to ensure excluded modules are not wrapped, but children are if param size is greater than ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:17.8935468Z   test_auto_wrap_preset_force_leaf (__main__.TestAutoWrap)
2022-11-23T03:12:17.8936206Z Test to ensure force-leaf modules are not wrapped, and children are not wrapped. The ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:17.8937044Z   test_auto_wrap_preset_force_leaf_custom (__main__.TestAutoWrap)
2022-11-23T03:12:17.8937749Z Test to ensure force-leaf modules are not wrapped. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:17.8938778Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=False)_use_device_id_False (__main__.TestAutoWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:12:17.8940067Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=False)_use_device_id_True (__main__.TestAutoWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:12:17.8941218Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=True)_use_device_id_False (__main__.TestAutoWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:12:17.8942367Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=True)_use_device_id_True (__main__.TestAutoWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:12:17.8943508Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=False)_use_device_id_False (__main__.TestAutoWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:12:17.8944634Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=False)_use_device_id_True (__main__.TestAutoWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:12:17.8945868Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=True)_use_device_id_False (__main__.TestAutoWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:12:17.8947002Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=True)_use_device_id_True (__main__.TestAutoWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:12:17.8948024Z   test_auto_wrap_with_ignored_modules_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:12:17.8948968Z   test_auto_wrap_with_ignored_modules_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:12:17.8949524Z   test_module_wrap_policy (__main__.TestAutoWrap)
2022-11-23T03:12:17.8950204Z Tests the ``ModuleWrapPolicy``. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:17.8950709Z   test_transformer_auto_wrap_policy (__main__.TestAutoWrap)
2022-11-23T03:12:17.8951366Z Tests the ``transformer_auto_wrap_policy``. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:17.8952188Z   test_wrap_disabled_outside_context (__main__.TestAutoWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:17.8953000Z   test_wrap_override_defaults (__main__.TestAutoWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:17.8953850Z   test_wrap_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:17.8954716Z   test_wrap_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:17.8955255Z   test_bn_always_wrapped_individually (__main__.TestFSDPWrap)
2022-11-23T03:12:17.8956024Z Ensures that by using _or_policy with _wrap_batchnorm_individually, even ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:12:17.8956641Z   test_error_already_wrapped_nested_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap)
2022-11-23T03:12:17.8957419Z Test that an error is raised if we attempt to wrap when submodules are ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:17.8958025Z   test_error_already_wrapped_nested_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap)
2022-11-23T03:12:17.8958777Z Test that an error is raised if we attempt to wrap when submodules are ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:17.8959378Z   test_error_already_wrapped_nested_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap)
2022-11-23T03:12:17.8960142Z Test that an error is raised if we attempt to wrap when submodules are ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:17.8960798Z   test_error_already_wrapped_nested_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap)
2022-11-23T03:12:17.8961560Z Test that an error is raised if we attempt to wrap when submodules are ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:17.8962683Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:12:17.8964011Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:12:17.8965321Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:12:17.8966622Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:12:17.8967927Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:12:17.8969218Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:12:17.8970519Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:12:17.8971878Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:12:17.8973177Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:12:17.8974476Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:12:17.8975786Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:12:17.8977352Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:12:17.8978660Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:12:17.8979965Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:12:17.8981272Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:12:17.8982567Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:12:17.8983624Z   test_wrap_batchnorm_individually_use_or_policy_False (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:17.8984506Z   test_wrap_batchnorm_individually_use_or_policy_True (__main__.TestFSDPWrap) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:17.8984897Z 
2022-11-23T03:12:17.8985166Z ----------------------------------------------------------------------
2022-11-23T03:12:17.8985498Z Ran 47 tests in 1.758s
2022-11-23T03:12:17.8985665Z 
2022-11-23T03:12:17.8985773Z OK (skipped=47)
2022-11-23T03:12:17.8985911Z 
2022-11-23T03:12:17.8986036Z Generating XML reports...
2022-11-23T03:12:17.8986607Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestAutoWrap-20221123031215.xml
2022-11-23T03:12:17.8987319Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestFSDPWrap-20221123031215.xml
2022-11-23T03:12:17.8987663Z 
2022-11-23T03:12:17.8988081Z ##[endgroup]
2022-11-23T03:12:17.8988673Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_wrap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_wrap_o2uqfxjr)
2022-11-23T03:12:17.8989015Z 
2022-11-23T03:12:18.2261498Z 
2022-11-23T03:12:18.2262170Z real	0m9.687s
2022-11-23T03:12:18.2262373Z user	0m18.201s
2022-11-23T03:12:18.2262638Z sys	0m15.947s
2022-11-23T03:12:18.2263258Z + python test/run_test.py --verbose -i distributed/checkpoint/test_checkpoint
2022-11-23T03:12:20.6282284Z Ignoring disabled issues:  []
2022-11-23T03:12:20.6829537Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:12:20.6830122Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:12:20.6830850Z Selected tests:
2022-11-23T03:12:20.6831139Z  distributed/checkpoint/test_checkpoint
2022-11-23T03:12:20.6856291Z Prioritized test from test file changes.
2022-11-23T03:12:20.6857719Z reordering tests for PR:
2022-11-23T03:12:20.6858097Z prioritized: []
2022-11-23T03:12:20.6859136Z the rest: ['distributed/checkpoint/test_checkpoint']
2022-11-23T03:12:20.6859368Z 
2022-11-23T03:12:20.6859924Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:12:20.6860846Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:12:20.6866464Z parallel (file granularity) tests:
2022-11-23T03:12:20.6866788Z  
2022-11-23T03:12:20.6867039Z serial (file granularity) tests:
2022-11-23T03:12:20.6867335Z  distributed/checkpoint/test_checkpoint
2022-11-23T03:12:23.0155749Z Ignoring disabled issues:  []
2022-11-23T03:12:23.0281346Z Ignoring disabled issues:  []
2022-11-23T03:12:23.5127597Z Running distributed/checkpoint/test_checkpoint ... [2022-11-23 03:12:23.512144]
2022-11-23T03:12:23.5128801Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:12:23.512614]
2022-11-23T03:12:27.7005045Z 
2022-11-23T03:12:27.7006004Z Expand the folded group to see the log file of distributed/checkpoint/test_checkpoint
2022-11-23T03:12:27.7007581Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_checkpoint_whxaem_6)
2022-11-23T03:12:27.7008042Z 
2022-11-23T03:12:27.7008164Z Running tests...
2022-11-23T03:12:27.7008739Z ----------------------------------------------------------------------
2022-11-23T03:12:27.7009305Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_checkpoint
2022-11-23T03:12:27.7009914Z   test_default_metadata (__main__.TestDistributedCheckpointing) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:12:27.7010580Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.757s)
2022-11-23T03:12:27.7011444Z   test_tensor_metadata_with_missing_rank_spec (__main__.TestDistributedCheckpointing) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:27.7012325Z   test_dummy_reader_works (__main__.TestDistributedFailure) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:27.7013182Z   test_dummy_writer_works (__main__.TestDistributedFailure) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:27.7014040Z   test_load_error_handling (__main__.TestDistributedFailure) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:27.7015185Z   test_load_error_handling_no_dist (__main__.TestDistributedFailure) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:27.7016082Z   test_save_error_handling (__main__.TestDistributedFailure) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:27.7017219Z   test_save_error_handling_no_dist (__main__.TestDistributedFailure) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:27.7017728Z 
2022-11-23T03:12:27.7018014Z ----------------------------------------------------------------------
2022-11-23T03:12:27.7018365Z Ran 8 tests in 1.765s
2022-11-23T03:12:27.7018429Z 
2022-11-23T03:12:27.7018658Z OK (skipped=8)
2022-11-23T03:12:27.7018784Z 
2022-11-23T03:12:27.7018909Z Generating XML reports...
2022-11-23T03:12:27.7019625Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_checkpoint/TEST-TestDistributedCheckpointing-20221123031225.xml
2022-11-23T03:12:27.7020519Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_checkpoint/TEST-TestDistributedFailure-20221123031225.xml
2022-11-23T03:12:27.7020902Z 
2022-11-23T03:12:27.7021209Z ##[endgroup]
2022-11-23T03:12:27.7021848Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_checkpoint_whxaem_6)
2022-11-23T03:12:27.7022233Z 
2022-11-23T03:12:28.0600467Z 
2022-11-23T03:12:28.0600915Z real	0m9.833s
2022-11-23T03:12:28.0601238Z user	0m17.715s
2022-11-23T03:12:28.0601487Z sys	0m15.241s
2022-11-23T03:12:28.0602081Z + python test/run_test.py --verbose -i distributed/checkpoint/test_file_system_checkpoint
2022-11-23T03:12:30.4381019Z Ignoring disabled issues:  []
2022-11-23T03:12:30.4926819Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:12:30.4927872Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:12:30.4928345Z Selected tests:
2022-11-23T03:12:30.4928666Z  distributed/checkpoint/test_file_system_checkpoint
2022-11-23T03:12:30.4952396Z Prioritized test from test file changes.
2022-11-23T03:12:30.4953015Z reordering tests for PR:
2022-11-23T03:12:30.4953493Z prioritized: []
2022-11-23T03:12:30.4954394Z the rest: ['distributed/checkpoint/test_file_system_checkpoint']
2022-11-23T03:12:30.4954870Z 
2022-11-23T03:12:30.4955916Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:12:30.4957726Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:12:30.4964271Z parallel (file granularity) tests:
2022-11-23T03:12:30.4964826Z  
2022-11-23T03:12:30.4965270Z serial (file granularity) tests:
2022-11-23T03:12:30.4965936Z  distributed/checkpoint/test_file_system_checkpoint
2022-11-23T03:12:32.7270822Z Ignoring disabled issues:  []
2022-11-23T03:12:32.8030233Z Ignoring disabled issues:  []
2022-11-23T03:12:33.2215224Z Running distributed/checkpoint/test_file_system_checkpoint ... [2022-11-23 03:12:33.220927]
2022-11-23T03:12:33.2218397Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:12:33.221402]
2022-11-23T03:12:37.4151141Z 
2022-11-23T03:12:37.4151934Z Expand the folded group to see the log file of distributed/checkpoint/test_file_system_checkpoint
2022-11-23T03:12:37.4153249Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_file_system_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_file_system_checkpoint_4u35bkoo)
2022-11-23T03:12:37.4153700Z 
2022-11-23T03:12:37.4153825Z Running tests...
2022-11-23T03:12:37.4154351Z ----------------------------------------------------------------------
2022-11-23T03:12:37.4154954Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint
2022-11-23T03:12:37.4155528Z   test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:12:37.4156222Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.760s)
2022-11-23T03:12:37.4157031Z   test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:12:37.4158244Z   test_save_load_bytes (__main__.TestDistributedReshardOnLoad) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:37.4159164Z   test_switch_between_sharded_tensor_to_tensor (__main__.TestDistributedReshardOnLoad) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:12:37.4160221Z   test_read_write_only_tensor (__main__.TestDistributedStateDictSaveLoad) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:37.4161495Z   test_read_write_shard_tensor (__main__.TestDistributedStateDictSaveLoadWithSharedTensor) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:37.4161954Z 
2022-11-23T03:12:37.4162221Z ----------------------------------------------------------------------
2022-11-23T03:12:37.4162555Z Ran 6 tests in 1.769s
2022-11-23T03:12:37.4162719Z 
2022-11-23T03:12:37.4162827Z OK (skipped=6)
2022-11-23T03:12:37.4162988Z 
2022-11-23T03:12:37.4163097Z Generating XML reports...
2022-11-23T03:12:37.4163783Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedReshardOnLoad-20221123031235.xml
2022-11-23T03:12:37.4165038Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoad-20221123031235.xml
2022-11-23T03:12:37.4166054Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20221123031235.xml
2022-11-23T03:12:37.4166544Z 
2022-11-23T03:12:37.4166843Z ##[endgroup]
2022-11-23T03:12:37.4167676Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_file_system_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_file_system_checkpoint_4u35bkoo)
2022-11-23T03:12:37.4168260Z 
2022-11-23T03:12:37.7512663Z 
2022-11-23T03:12:37.7513456Z real	0m9.691s
2022-11-23T03:12:37.7513721Z user	0m17.465s
2022-11-23T03:12:37.7513977Z sys	0m15.857s
2022-11-23T03:12:37.7514503Z + python test/run_test.py --verbose -i distributed/_shard/sharding_spec/test_sharding_spec
2022-11-23T03:12:40.1489780Z Ignoring disabled issues:  []
2022-11-23T03:12:40.2034040Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:12:40.2034599Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:12:40.2034964Z Selected tests:
2022-11-23T03:12:40.2035281Z  distributed/_shard/sharding_spec/test_sharding_spec
2022-11-23T03:12:40.2059158Z Prioritized test from test file changes.
2022-11-23T03:12:40.2059539Z reordering tests for PR:
2022-11-23T03:12:40.2059816Z prioritized: []
2022-11-23T03:12:40.2060346Z the rest: ['distributed/_shard/sharding_spec/test_sharding_spec']
2022-11-23T03:12:40.2060566Z 
2022-11-23T03:12:40.2061535Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:12:40.2062531Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:12:40.2067472Z parallel (file granularity) tests:
2022-11-23T03:12:40.2067764Z  
2022-11-23T03:12:40.2067994Z serial (file granularity) tests:
2022-11-23T03:12:40.2068347Z  distributed/_shard/sharding_spec/test_sharding_spec
2022-11-23T03:12:42.5530435Z Ignoring disabled issues:  []
2022-11-23T03:12:42.5531170Z Ignoring disabled issues:  []
2022-11-23T03:12:43.0265759Z Running distributed/_shard/sharding_spec/test_sharding_spec ... [2022-11-23 03:12:43.025888]
2022-11-23T03:12:43.0268320Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharding_spec/test_sharding_spec.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:12:43.026383]
2022-11-23T03:12:47.1413481Z 
2022-11-23T03:12:47.1414669Z Expand the folded group to see the log file of distributed/_shard/sharding_spec/test_sharding_spec
2022-11-23T03:12:47.1416001Z ##[group]PRINTING LOG FILE of distributed/_shard/sharding_spec/test_sharding_spec (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharding_spec-test_sharding_spec_ms7ab14v)
2022-11-23T03:12:47.1416676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp84u_po38
2022-11-23T03:12:47.1417570Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp84u_po38/_remote_module_non_scriptable.py
2022-11-23T03:12:47.1417859Z 
2022-11-23T03:12:47.1417952Z Running tests...
2022-11-23T03:12:47.1418499Z ----------------------------------------------------------------------
2022-11-23T03:12:47.1419100Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec
2022-11-23T03:12:47.1419670Z   test_custom_sharding_spec (__main__.TestCustomShardingSpec) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:12:47.1420345Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.671s)
2022-11-23T03:12:47.1420819Z   test_custom_sharding_spec_shard_tensor (__main__.TestCustomShardingSpec)
2022-11-23T03:12:47.1421509Z Test custom spec can be invoked from the ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:47.1422044Z   test_custom_sharding_spec_tensor_ctor (__main__.TestCustomShardingSpec)
2022-11-23T03:12:47.1422750Z Test sharded_tensor.ones(...) with the custom ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:47.1423552Z   test_check_overlapping (__main__.TestShardingSpec) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:12:47.1424382Z   test_chunked_sharding_spec (__main__.TestShardingSpec) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:12:47.1425215Z   test_device_placement (__main__.TestShardingSpec) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:47.1426061Z   test_enumerable_sharding_spec (__main__.TestShardingSpec) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.004s)
2022-11-23T03:12:47.1426891Z   test_get_chunk_sharding_params (__main__.TestShardingSpec) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:47.1427872Z   test_get_chunked_dim_size (__main__.TestShardingSpec) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:47.1428709Z   test_get_split_size (__main__.TestShardingSpec) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:47.1429580Z   test_infer_sharding_spec_from_shards_metadata (__main__.TestShardingSpec) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:47.1429976Z 
2022-11-23T03:12:47.1430247Z ----------------------------------------------------------------------
2022-11-23T03:12:47.1430560Z Ran 11 tests in 1.687s
2022-11-23T03:12:47.1430844Z 
2022-11-23T03:12:47.1430844Z OK (skipped=11)
2022-11-23T03:12:47.1431005Z 
2022-11-23T03:12:47.1431149Z Generating XML reports...
2022-11-23T03:12:47.1431816Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20221123031245.xml
2022-11-23T03:12:47.1432642Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20221123031245.xml
2022-11-23T03:12:47.1433092Z 
2022-11-23T03:12:47.1433414Z ##[endgroup]
2022-11-23T03:12:47.1434095Z FINISHED PRINTING LOG FILE of distributed/_shard/sharding_spec/test_sharding_spec (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharding_spec-test_sharding_spec_ms7ab14v)
2022-11-23T03:12:47.1434504Z 
2022-11-23T03:12:47.4995560Z 
2022-11-23T03:12:47.4995991Z real	0m9.748s
2022-11-23T03:12:47.4996367Z user	0m17.294s
2022-11-23T03:12:47.4996623Z sys	0m14.446s
2022-11-23T03:12:47.4997272Z + python test/run_test.py --verbose -i distributed/_shard/sharding_plan/test_sharding_plan
2022-11-23T03:12:49.8595503Z Ignoring disabled issues:  []
2022-11-23T03:12:49.9132900Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:12:49.9133490Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:12:49.9133840Z Selected tests:
2022-11-23T03:12:49.9134161Z  distributed/_shard/sharding_plan/test_sharding_plan
2022-11-23T03:12:49.9161038Z Prioritized test from test file changes.
2022-11-23T03:12:49.9161362Z reordering tests for PR:
2022-11-23T03:12:49.9161639Z prioritized: []
2022-11-23T03:12:49.9162168Z the rest: ['distributed/_shard/sharding_plan/test_sharding_plan']
2022-11-23T03:12:49.9162415Z 
2022-11-23T03:12:49.9162962Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:12:49.9163882Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:12:49.9170257Z parallel (file granularity) tests:
2022-11-23T03:12:49.9170745Z  
2022-11-23T03:12:49.9171068Z serial (file granularity) tests:
2022-11-23T03:12:49.9171436Z  distributed/_shard/sharding_plan/test_sharding_plan
2022-11-23T03:12:52.2648588Z Ignoring disabled issues:  []
2022-11-23T03:12:52.2883778Z Ignoring disabled issues:  []
2022-11-23T03:12:52.7699077Z Running distributed/_shard/sharding_plan/test_sharding_plan ... [2022-11-23 03:12:52.769195]
2022-11-23T03:12:52.7699956Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharding_plan/test_sharding_plan.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:12:52.769636]
2022-11-23T03:12:56.9555753Z 
2022-11-23T03:12:56.9556402Z Expand the folded group to see the log file of distributed/_shard/sharding_plan/test_sharding_plan
2022-11-23T03:12:56.9557515Z ##[group]PRINTING LOG FILE of distributed/_shard/sharding_plan/test_sharding_plan (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharding_plan-test_sharding_plan_hccur5bl)
2022-11-23T03:12:56.9557935Z 
2022-11-23T03:12:56.9558046Z Running tests...
2022-11-23T03:12:56.9558895Z ----------------------------------------------------------------------
2022-11-23T03:12:56.9559521Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharding_plan.test_sharding_plan
2022-11-23T03:12:56.9560072Z   test_custom_sharding_planner (__main__.TestShardingPlan) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:12:56.9560731Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.757s)
2022-11-23T03:12:56.9561650Z   test_reshard_to_ddp_sharding_plan (__main__.TestShardingPlan) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:12:56.9562469Z   test_shard_module_sub_process_group (__main__.TestShardingPlan) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:12:56.9563296Z   test_sharding_plan_errors (__main__.TestShardingPlan) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:12:56.9564263Z   test_sharding_plan_simple_megatron (__main__.TestShardingPlan) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.004s)
2022-11-23T03:12:56.9564637Z 
2022-11-23T03:12:56.9564905Z ----------------------------------------------------------------------
2022-11-23T03:12:56.9565404Z Ran 5 tests in 1.766s
2022-11-23T03:12:56.9565550Z 
2022-11-23T03:12:56.9565661Z OK (skipped=5)
2022-11-23T03:12:56.9565816Z 
2022-11-23T03:12:56.9565954Z Generating XML reports...
2022-11-23T03:12:56.9566584Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_plan.test_sharding_plan/TEST-TestShardingPlan-20221123031254.xml
2022-11-23T03:12:56.9566953Z 
2022-11-23T03:12:56.9567241Z ##[endgroup]
2022-11-23T03:12:56.9567914Z FINISHED PRINTING LOG FILE of distributed/_shard/sharding_plan/test_sharding_plan (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharding_plan-test_sharding_plan_hccur5bl)
2022-11-23T03:12:56.9568315Z 
2022-11-23T03:12:57.3139328Z 
2022-11-23T03:12:57.3139514Z real	0m9.814s
2022-11-23T03:12:57.3139779Z user	0m17.768s
2022-11-23T03:12:57.3140021Z sys	0m15.907s
2022-11-23T03:12:57.3140519Z + python test/run_test.py --verbose -i distributed/_shard/sharded_tensor/test_megatron_prototype
2022-11-23T03:12:59.6726943Z Ignoring disabled issues:  []
2022-11-23T03:12:59.7269488Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:12:59.7270661Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:12:59.7271213Z Selected tests:
2022-11-23T03:12:59.7271538Z  distributed/_shard/sharded_tensor/test_megatron_prototype
2022-11-23T03:12:59.7295669Z Prioritized test from test file changes.
2022-11-23T03:12:59.7296248Z reordering tests for PR:
2022-11-23T03:12:59.7297295Z prioritized: []
2022-11-23T03:12:59.7298019Z the rest: ['distributed/_shard/sharded_tensor/test_megatron_prototype']
2022-11-23T03:12:59.7298278Z 
2022-11-23T03:12:59.7298795Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:12:59.7299738Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:12:59.7303133Z parallel (file granularity) tests:
2022-11-23T03:12:59.7303742Z  
2022-11-23T03:12:59.7304216Z serial (file granularity) tests:
2022-11-23T03:12:59.7304616Z  distributed/_shard/sharded_tensor/test_megatron_prototype
2022-11-23T03:13:02.0575014Z Ignoring disabled issues:  []
2022-11-23T03:13:02.0608491Z Ignoring disabled issues:  []
2022-11-23T03:13:02.4948266Z Running distributed/_shard/sharded_tensor/test_megatron_prototype ... [2022-11-23 03:13:02.494117]
2022-11-23T03:13:02.4950133Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_megatron_prototype.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:13:02.494635]
2022-11-23T03:13:06.6531839Z 
2022-11-23T03:13:06.6532995Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/test_megatron_prototype
2022-11-23T03:13:06.6534147Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_megatron_prototype (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_megatron_prototype_kg8wxlht)
2022-11-23T03:13:06.6534593Z 
2022-11-23T03:13:06.6534697Z Running tests...
2022-11-23T03:13:06.6535281Z ----------------------------------------------------------------------
2022-11-23T03:13:06.6535910Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype
2022-11-23T03:13:06.6536429Z   test_megatron_two_layer_prototype (__main__.TestShardedTensorMegatronLinear) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:13:06.6537840Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.765s)
2022-11-23T03:13:06.6538109Z 
2022-11-23T03:13:06.6538446Z ----------------------------------------------------------------------
2022-11-23T03:13:06.6538756Z Ran 1 test in 1.765s
2022-11-23T03:13:06.6538895Z 
2022-11-23T03:13:06.6539050Z OK (skipped=1)
2022-11-23T03:13:06.6539190Z 
2022-11-23T03:13:06.6539323Z Generating XML reports...
2022-11-23T03:13:06.6540047Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype/TEST-TestShardedTensorMegatronLinear-20221123031304.xml
2022-11-23T03:13:06.6540465Z 
2022-11-23T03:13:06.6540686Z ##[endgroup]
2022-11-23T03:13:06.6541376Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_megatron_prototype (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_megatron_prototype_kg8wxlht)
2022-11-23T03:13:06.6541806Z 
2022-11-23T03:13:07.0328152Z 
2022-11-23T03:13:07.0328849Z real	0m9.719s
2022-11-23T03:13:07.0329113Z user	0m16.812s
2022-11-23T03:13:07.0329379Z sys	0m12.893s
2022-11-23T03:13:07.0329980Z + python test/run_test.py --verbose -i distributed/_shard/sharded_tensor/test_sharded_tensor
2022-11-23T03:13:09.4393593Z Ignoring disabled issues:  []
2022-11-23T03:13:09.4940406Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:13:09.4940996Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:13:09.4941373Z Selected tests:
2022-11-23T03:13:09.4941676Z  distributed/_shard/sharded_tensor/test_sharded_tensor
2022-11-23T03:13:09.4966126Z Prioritized test from test file changes.
2022-11-23T03:13:09.4966445Z reordering tests for PR:
2022-11-23T03:13:09.4966776Z prioritized: []
2022-11-23T03:13:09.4967331Z the rest: ['distributed/_shard/sharded_tensor/test_sharded_tensor']
2022-11-23T03:13:09.4967583Z 
2022-11-23T03:13:09.4968094Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:13:09.4969035Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:13:09.4974026Z parallel (file granularity) tests:
2022-11-23T03:13:09.4974319Z  
2022-11-23T03:13:09.4974574Z serial (file granularity) tests:
2022-11-23T03:13:09.4974915Z  distributed/_shard/sharded_tensor/test_sharded_tensor
2022-11-23T03:13:11.7954638Z Ignoring disabled issues:  []
2022-11-23T03:13:11.8566593Z Ignoring disabled issues:  []
2022-11-23T03:13:12.2307544Z Running distributed/_shard/sharded_tensor/test_sharded_tensor ... [2022-11-23 03:13:12.230091]
2022-11-23T03:13:12.2309081Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_sharded_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:13:12.230555]
2022-11-23T03:13:30.6400316Z 
2022-11-23T03:13:30.6401024Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/test_sharded_tensor
2022-11-23T03:13:30.6402077Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_sharded_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_sharded_tensor_obx4uoon)
2022-11-23T03:13:30.6405679Z 
2022-11-23T03:13:30.6406032Z Running tests...
2022-11-23T03:13:30.6406646Z ----------------------------------------------------------------------
2022-11-23T03:13:30.6407364Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor
2022-11-23T03:13:30.6408020Z   test_empty (__main__.TestCreateTensorFromParams) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:13:30.6408887Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.718s)
2022-11-23T03:13:30.6409980Z   test_local_tensor (__main__.TestLocalTensor) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:30.6410817Z   test_local_tensor_error (__main__.TestLocalTensor) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:30.6411632Z   test_collect_local_shard (__main__.TestModuleHookApi) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:30.6412470Z   test_reshard_output (__main__.TestModuleHookApi) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:30.6413344Z   test_create_shard_with_no_placement (__main__.TestShardMetadata) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:30.6414195Z   test_shard_metadata_init (__main__.TestShardMetadata) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:30.6415030Z   test_shard_parameter (__main__.TestShardParameter) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:30.6415867Z   test_shard_parameter_errors (__main__.TestShardParameter) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:13:30.6417116Z   test_shard_tensor (__main__.TestShardTensor) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:30.6417999Z   test_shard_tensor_errors (__main__.TestShardTensor) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:13:30.6418838Z   test_cleanup (__main__.TestShardedTensorChunked) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:30.6419678Z   test_complete_world_size (__main__.TestShardedTensorChunked) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:13:30.6420280Z   test_create_sharded_tensor_like (__main__.TestShardedTensorChunked)
2022-11-23T03:13:30.6421544Z Test tensor like methods, i.e. torch.zeros_like(...), torch.full_like, etc. ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:13:30.6422836Z   test_create_sharded_tensor_with_full (__main__.TestShardedTensorChunked)
2022-11-23T03:13:30.6424170Z Test sharded_tensor.full(...) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:30.6425344Z   test_create_sharded_tensor_with_ones (__main__.TestShardedTensorChunked)
2022-11-23T03:13:30.6426032Z Test sharded_tensor.ones(...) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:30.6426562Z   test_create_sharded_tensor_with_rand (__main__.TestShardedTensorChunked)
2022-11-23T03:13:30.6427253Z Test sharded_tensor.rand(...)/randn(...) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:13:30.6427956Z   test_create_sharded_tensor_with_zeros (__main__.TestShardedTensorChunked)
2022-11-23T03:13:30.6429427Z Test sharded_tensor.zeros(...) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:30.6430057Z   test_gather_even (__main__.TestShardedTensorChunked)
2022-11-23T03:13:30.6430893Z Test _sharded_tensor.gather(...) with evenly distributed._shards ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:30.6431419Z   test_gather_uneven (__main__.TestShardedTensorChunked)
2022-11-23T03:13:30.6432130Z Test _sharded_tensor.gather(...) with unevenly distributed._shards ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:30.6433084Z   test_insufficient_sharding_dims (__main__.TestShardedTensorChunked) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:30.6433980Z   test_invalid_pg_rpc_ranks (__main__.TestShardedTensorChunked) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:30.6434830Z   test_invalid_sharding (__main__.TestShardedTensorChunked) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:13:30.6435717Z   test_load_state_dict_errors (__main__.TestShardedTensorChunked) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:13:30.6436595Z   test_multiple_local_shards (__main__.TestShardedTensorChunked) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:13:30.6437446Z   test_new_group (__main__.TestShardedTensorChunked) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:13:30.6438302Z   test_partial_world_size (__main__.TestShardedTensorChunked) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:13:30.6439163Z   test_sharded_tensor_metadata (__main__.TestShardedTensorChunked) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:13:30.6440047Z   test_sharded_tensor_sizes (__main__.TestShardedTensorChunked) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:13:30.6441029Z   test_sharding_columns (__main__.TestShardedTensorChunked) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:30.6441874Z   test_state_dict (__main__.TestShardedTensorChunked) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:30.6442766Z   test_state_dict_new_group (__main__.TestShardedTensorChunked) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:30.6443657Z   test_state_dict_no_sharded_tensors (__main__.TestShardedTensorChunked) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:30.6444518Z   test_custom_op (__main__.TestShardedTensorCustomOps) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:30.6445419Z   test_custom_op_errors (__main__.TestShardedTensorCustomOps) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:30.6446284Z   test_custom_op_override (__main__.TestShardedTensorCustomOps) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:30.6446860Z   test_create_sharded_tensor_with_ones (__main__.TestShardedTensorEnumerable)
2022-11-23T03:13:30.6447511Z Test sharded_tensor.ones(...) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:13:30.6448071Z   test_gather_even (__main__.TestShardedTensorEnumerable)
2022-11-23T03:13:30.6448780Z Test _sharded_tensor.gather(...) with evenly distributed._shards ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:30.6449320Z   test_gather_uneven (__main__.TestShardedTensorEnumerable)
2022-11-23T03:13:30.6450018Z Test _sharded_tensor.gather(...) with unevenly distributed._shards ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:13:30.6450875Z   test_grid_sharding (__main__.TestShardedTensorEnumerable) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:13:30.6451756Z   test_multiple_local_shards (__main__.TestShardedTensorEnumerable) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:13:30.6452617Z   test_new_group (__main__.TestShardedTensorEnumerable) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:13:30.6453480Z   test_partial_world_size (__main__.TestShardedTensorEnumerable) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:13:30.6454350Z   test_sharded_tensor_device (__main__.TestShardedTensorEnumerable) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:30.6455244Z   test_sharded_tensor_metadata (__main__.TestShardedTensorEnumerable) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:13:30.6456131Z   test_sharded_tensor_to_cpu (__main__.TestShardedTensorEnumerable) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:13:30.6457341Z   test_sharded_tensor_to_cuda (__main__.TestShardedTensorEnumerable) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:13:30.6458226Z   test_sharded_tensor_to_test (__main__.TestShardedTensorEnumerable) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:13:30.6459084Z   test_uneven_shards (__main__.TestShardedTensorEnumerable) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:13:30.6460089Z   test_with_rpc_names (__main__.TestShardedTensorEnumerable) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:13:30.6460805Z   test_init_from_local_shards (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25632
2022-11-23T03:13:30.6461371Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25633
2022-11-23T03:13:30.6461801Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25634
2022-11-23T03:13:30.6462236Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25635
2022-11-23T03:13:30.6462854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:13:30.6463291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:13:30.6463867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:13:30.6464340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:13:30.6464923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:13:30.6465432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:13:30.6466008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:13:30.6466471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:13:30.6467048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:13:30.6467472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:13:30.6468044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:13:30.6468504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:13:30.6469067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:13:30.6469518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:13:30.6470088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:13:30.6470550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:13:30.6470973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:13:30.6471445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:13:30.6484932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:13:30.6485432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:13:30.6485951Z fi_getinfo: -61
2022-11-23T03:13:30.6486231Z fi_getinfo: -61
2022-11-23T03:13:30.6486516Z fi_getinfo: -61
2022-11-23T03:13:30.6486770Z fi_getinfo: -61
2022-11-23T03:13:30.6487157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:13:30.6487658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:13:30.6488140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:13:30.6488806Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:13:30.6489348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:13:30.6490011Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:13:30.6490791Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:13:30.6491495Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:13:30.6491909Z ok (14.145s)
2022-11-23T03:13:30.6492650Z   test_init_from_local_shards_and_global_metadata (__main__.TestShardedTensorFromLocalShards) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:13:30.6493662Z   test_init_from_local_shards_and_global_metadata_invalid_shards (__main__.TestShardedTensorFromLocalShards) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:13:30.6494693Z   test_init_from_local_shards_invalid_local_shards (__main__.TestShardedTensorFromLocalShards) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:13:30.6495687Z   test_init_from_local_shards_invalid_pin_memory (__main__.TestShardedTensorFromLocalShards) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:30.6497081Z   test_init_from_local_shards_invalid_property_cross_ranks (__main__.TestShardedTensorFromLocalShards) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:13:30.6498106Z   test_init_from_local_shards_invalid_shards_gaps (__main__.TestShardedTensorFromLocalShards) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:30.6499109Z   test_init_from_local_shards_invalid_shards_overlap (__main__.TestShardedTensorFromLocalShards) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:30.6500064Z   test_init_from_local_shards_new_group (__main__.TestShardedTensorFromLocalShards) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:13:30.6500991Z   test_local_shards (__main__.TestShardedTensorFromLocalShards) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:30.6501967Z   test_st_base_init_from_local_shards_and_global_metadata (__main__.TestShardedTensorFromLocalShards) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:13:30.6502934Z   test_init_from_local_tensor (__main__.TestShardedTensorFromLocalTensor) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:30.6503880Z   test_init_from_local_tensor_errors (__main__.TestShardedTensorFromLocalTensor) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:30.6504788Z   test_serialize_and_deserialize (__main__.TestShardedTensorMetadata) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:13:30.6505193Z 
2022-11-23T03:13:30.6505473Z ----------------------------------------------------------------------
2022-11-23T03:13:30.6505820Z Ran 64 tests in 15.970s
2022-11-23T03:13:30.6505991Z 
2022-11-23T03:13:30.6506110Z OK (skipped=63)
2022-11-23T03:13:30.6506247Z 
2022-11-23T03:13:30.6506380Z Generating XML reports...
2022-11-23T03:13:30.6507094Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalShards-20221123031314.xml
2022-11-23T03:13:30.6508016Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestCreateTensorFromParams-20221123031314.xml
2022-11-23T03:13:30.6508971Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestLocalTensor-20221123031314.xml
2022-11-23T03:13:30.6509789Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestModuleHookApi-20221123031314.xml
2022-11-23T03:13:30.6510599Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardMetadata-20221123031314.xml
2022-11-23T03:13:30.6511414Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardParameter-20221123031314.xml
2022-11-23T03:13:30.6512226Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardTensor-20221123031314.xml
2022-11-23T03:13:30.6513035Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorChunked-20221123031314.xml
2022-11-23T03:13:30.6513920Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorCustomOps-20221123031314.xml
2022-11-23T03:13:30.6514880Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorEnumerable-20221123031314.xml
2022-11-23T03:13:30.6515793Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalTensor-20221123031314.xml
2022-11-23T03:13:30.6516666Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorMetadata-20221123031314.xml
2022-11-23T03:13:30.6517063Z 
2022-11-23T03:13:30.6517420Z ##[endgroup]
2022-11-23T03:13:30.6518112Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_sharded_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_sharded_tensor_obx4uoon)
2022-11-23T03:13:30.6518530Z 
2022-11-23T03:13:31.0355272Z 
2022-11-23T03:13:31.0355481Z real	0m24.003s
2022-11-23T03:13:31.0355765Z user	1m2.995s
2022-11-23T03:13:31.0356016Z sys	0m56.075s
2022-11-23T03:13:31.0356650Z + python test/run_test.py --verbose -i distributed/_shard/sharded_tensor/test_sharded_tensor_reshard
2022-11-23T03:13:33.3857554Z Ignoring disabled issues:  []
2022-11-23T03:13:33.4425976Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:13:33.4427231Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:13:33.4427910Z Selected tests:
2022-11-23T03:13:33.4428228Z  distributed/_shard/sharded_tensor/test_sharded_tensor_reshard
2022-11-23T03:13:33.4449394Z Prioritized test from test file changes.
2022-11-23T03:13:33.4450130Z reordering tests for PR:
2022-11-23T03:13:33.4450695Z prioritized: []
2022-11-23T03:13:33.4451566Z the rest: ['distributed/_shard/sharded_tensor/test_sharded_tensor_reshard']
2022-11-23T03:13:33.4451826Z 
2022-11-23T03:13:33.4452381Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:13:33.4453335Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:13:33.4456188Z parallel (file granularity) tests:
2022-11-23T03:13:33.4456489Z  
2022-11-23T03:13:33.4457097Z serial (file granularity) tests:
2022-11-23T03:13:33.4458301Z  distributed/_shard/sharded_tensor/test_sharded_tensor_reshard
2022-11-23T03:13:35.7618501Z Ignoring disabled issues:  []
2022-11-23T03:13:35.7949409Z Ignoring disabled issues:  []
2022-11-23T03:13:36.1517771Z Running distributed/_shard/sharded_tensor/test_sharded_tensor_reshard ... [2022-11-23 03:13:36.151112]
2022-11-23T03:13:36.1519611Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_sharded_tensor_reshard.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:13:36.151591]
2022-11-23T03:13:40.2402270Z 
2022-11-23T03:13:40.2403178Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/test_sharded_tensor_reshard
2022-11-23T03:13:40.2404279Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_sharded_tensor_reshard (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_sharded_tensor_reshard_vvhac6kq)
2022-11-23T03:13:40.2404742Z 
2022-11-23T03:13:40.2404864Z Running tests...
2022-11-23T03:13:40.2405418Z ----------------------------------------------------------------------
2022-11-23T03:13:40.2406018Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard
2022-11-23T03:13:40.2406482Z   test_sharded_tensor_reshard (__main__.TestReshard) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:13:40.2407130Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.715s)
2022-11-23T03:13:40.2408252Z   test_sharded_tensor_reshard_errors (__main__.TestReshard) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:40.2408622Z 
2022-11-23T03:13:40.2408889Z ----------------------------------------------------------------------
2022-11-23T03:13:40.2409200Z Ran 2 tests in 1.717s
2022-11-23T03:13:40.2409364Z 
2022-11-23T03:13:40.2409472Z OK (skipped=2)
2022-11-23T03:13:40.2409630Z 
2022-11-23T03:13:40.2409755Z Generating XML reports...
2022-11-23T03:13:40.2410388Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard/TEST-TestReshard-20221123031338.xml
2022-11-23T03:13:40.2410732Z 
2022-11-23T03:13:40.2411047Z ##[endgroup]
2022-11-23T03:13:40.2411759Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_sharded_tensor_reshard (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_sharded_tensor_reshard_vvhac6kq)
2022-11-23T03:13:40.2412186Z 
2022-11-23T03:13:40.5748069Z 
2022-11-23T03:13:40.5748389Z real	0m9.539s
2022-11-23T03:13:40.5748819Z user	0m17.746s
2022-11-23T03:13:40.5749590Z sys	0m16.003s
2022-11-23T03:13:40.5750345Z + python test/run_test.py --verbose -i distributed/_shard/sharded_tensor/ops/test_chunk
2022-11-23T03:13:42.9451450Z Ignoring disabled issues:  []
2022-11-23T03:13:42.9985333Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:13:42.9985922Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:13:42.9986290Z Selected tests:
2022-11-23T03:13:42.9986614Z  distributed/_shard/sharded_tensor/ops/test_chunk
2022-11-23T03:13:43.0012655Z Prioritized test from test file changes.
2022-11-23T03:13:43.0013020Z reordering tests for PR:
2022-11-23T03:13:43.0013312Z prioritized: []
2022-11-23T03:13:43.0013833Z the rest: ['distributed/_shard/sharded_tensor/ops/test_chunk']
2022-11-23T03:13:43.0014081Z 
2022-11-23T03:13:43.0014626Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:13:43.0015584Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:13:43.0022637Z parallel (file granularity) tests:
2022-11-23T03:13:43.0022953Z  
2022-11-23T03:13:43.0023203Z serial (file granularity) tests:
2022-11-23T03:13:43.0023539Z  distributed/_shard/sharded_tensor/ops/test_chunk
2022-11-23T03:13:45.2983670Z Ignoring disabled issues:  []
2022-11-23T03:13:45.2983998Z Ignoring disabled issues:  []
2022-11-23T03:13:45.7751450Z Running distributed/_shard/sharded_tensor/ops/test_chunk ... [2022-11-23 03:13:45.774474]
2022-11-23T03:13:45.7753831Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_chunk.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:13:45.774976]
2022-11-23T03:13:49.8731141Z 
2022-11-23T03:13:49.8731769Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_chunk
2022-11-23T03:13:49.8732857Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_chunk (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_chunk_pfz4t3md)
2022-11-23T03:13:49.8733277Z 
2022-11-23T03:13:49.8733404Z Running tests...
2022-11-23T03:13:49.8733972Z ----------------------------------------------------------------------
2022-11-23T03:13:49.8734570Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk
2022-11-23T03:13:49.8735154Z   test_sharded_chunk (__main__.TestShardedTensorChunkOps) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:13:49.8736122Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.715s)
2022-11-23T03:13:49.8737357Z   test_sharded_chunk_error (__main__.TestShardedTensorChunkOps) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:49.8737789Z 
2022-11-23T03:13:49.8738082Z ----------------------------------------------------------------------
2022-11-23T03:13:49.8738436Z Ran 2 tests in 1.716s
2022-11-23T03:13:49.8738609Z 
2022-11-23T03:13:49.8738731Z OK (skipped=2)
2022-11-23T03:13:49.8738897Z 
2022-11-23T03:13:49.8739007Z Generating XML reports...
2022-11-23T03:13:49.8739688Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk/TEST-TestShardedTensorChunkOps-20221123031347.xml
2022-11-23T03:13:49.8740089Z 
2022-11-23T03:13:49.8740432Z ##[endgroup]
2022-11-23T03:13:49.8741080Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_chunk (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_chunk_pfz4t3md)
2022-11-23T03:13:49.8741483Z 
2022-11-23T03:13:50.2681658Z 
2022-11-23T03:13:50.2682109Z real	0m9.693s
2022-11-23T03:13:50.2682443Z user	0m16.948s
2022-11-23T03:13:50.2682731Z sys	0m13.227s
2022-11-23T03:13:50.2683346Z + python test/run_test.py --verbose -i distributed/_shard/sharded_tensor/ops/test_elementwise_ops
2022-11-23T03:13:52.7112565Z Ignoring disabled issues:  []
2022-11-23T03:13:52.7655425Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:13:52.7656024Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:13:52.7656368Z Selected tests:
2022-11-23T03:13:52.7657088Z  distributed/_shard/sharded_tensor/ops/test_elementwise_ops
2022-11-23T03:13:52.7680774Z Prioritized test from test file changes.
2022-11-23T03:13:52.7681113Z reordering tests for PR:
2022-11-23T03:13:52.7681425Z prioritized: []
2022-11-23T03:13:52.7682003Z the rest: ['distributed/_shard/sharded_tensor/ops/test_elementwise_ops']
2022-11-23T03:13:52.7682261Z 
2022-11-23T03:13:52.7682813Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:13:52.7683747Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:13:52.7688448Z parallel (file granularity) tests:
2022-11-23T03:13:52.7688742Z  
2022-11-23T03:13:52.7689016Z serial (file granularity) tests:
2022-11-23T03:13:52.7689357Z  distributed/_shard/sharded_tensor/ops/test_elementwise_ops
2022-11-23T03:13:55.0207182Z Ignoring disabled issues:  []
2022-11-23T03:13:55.0231645Z Ignoring disabled issues:  []
2022-11-23T03:13:55.5106231Z Running distributed/_shard/sharded_tensor/ops/test_elementwise_ops ... [2022-11-23 03:13:55.509923]
2022-11-23T03:13:55.5107553Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_elementwise_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:13:55.510377]
2022-11-23T03:13:59.6018686Z 
2022-11-23T03:13:59.6019412Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_elementwise_ops
2022-11-23T03:13:59.6020518Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_elementwise_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_elementwise_ops_ema2wxem)
2022-11-23T03:13:59.6020958Z 
2022-11-23T03:13:59.6021073Z Running tests...
2022-11-23T03:13:59.6021642Z ----------------------------------------------------------------------
2022-11-23T03:13:59.6022303Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops
2022-11-23T03:13:59.6023184Z   test_sharded_dropout (__main__.TestShardedTensorElementWiseOps) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:13:59.6023857Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.703s)
2022-11-23T03:13:59.6024673Z   test_sharded_gelu (__main__.TestShardedTensorElementWiseOps) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:59.6025581Z   test_sharded_relu (__main__.TestShardedTensorElementWiseOps) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:59.6026500Z   test_sharded_tensor_nan_to_num (__main__.TestShardedTensorElementWiseOps) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:13:59.6026916Z 
2022-11-23T03:13:59.6027196Z ----------------------------------------------------------------------
2022-11-23T03:13:59.6027517Z Ran 4 tests in 1.707s
2022-11-23T03:13:59.6027690Z 
2022-11-23T03:13:59.6027808Z OK (skipped=4)
2022-11-23T03:13:59.6027987Z 
2022-11-23T03:13:59.6028122Z Generating XML reports...
2022-11-23T03:13:59.6028823Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops/TEST-TestShardedTensorElementWiseOps-20221123031357.xml
2022-11-23T03:13:59.6029259Z 
2022-11-23T03:13:59.6029577Z ##[endgroup]
2022-11-23T03:13:59.6030293Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_elementwise_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_elementwise_ops_ema2wxem)
2022-11-23T03:13:59.6030718Z 
2022-11-23T03:13:59.9901214Z 
2022-11-23T03:13:59.9901629Z real	0m9.722s
2022-11-23T03:13:59.9901955Z user	0m15.737s
2022-11-23T03:13:59.9902204Z sys	0m11.714s
2022-11-23T03:13:59.9902812Z + python test/run_test.py --verbose -i distributed/_shard/sharded_tensor/ops/test_embedding
2022-11-23T03:14:02.3716697Z Ignoring disabled issues:  []
2022-11-23T03:14:02.4261467Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:14:02.4262323Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:14:02.4262690Z Selected tests:
2022-11-23T03:14:02.4262993Z  distributed/_shard/sharded_tensor/ops/test_embedding
2022-11-23T03:14:02.4287881Z Prioritized test from test file changes.
2022-11-23T03:14:02.4288529Z reordering tests for PR:
2022-11-23T03:14:02.4289097Z prioritized: []
2022-11-23T03:14:02.4289777Z the rest: ['distributed/_shard/sharded_tensor/ops/test_embedding']
2022-11-23T03:14:02.4290106Z 
2022-11-23T03:14:02.4290977Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:14:02.4291963Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:14:02.4296023Z parallel (file granularity) tests:
2022-11-23T03:14:02.4296632Z  
2022-11-23T03:14:02.4297573Z serial (file granularity) tests:
2022-11-23T03:14:02.4297986Z  distributed/_shard/sharded_tensor/ops/test_embedding
2022-11-23T03:14:04.7811602Z Ignoring disabled issues:  []
2022-11-23T03:14:04.7922576Z Ignoring disabled issues:  []
2022-11-23T03:14:05.2033879Z Running distributed/_shard/sharded_tensor/ops/test_embedding ... [2022-11-23 03:14:05.202760]
2022-11-23T03:14:05.2035217Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_embedding.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:14:05.203200]
2022-11-23T03:14:09.4365739Z 
2022-11-23T03:14:09.4366887Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_embedding
2022-11-23T03:14:09.4368345Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_embedding (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_embedding_41x2jewk)
2022-11-23T03:14:09.4368765Z 
2022-11-23T03:14:09.4368921Z Running tests...
2022-11-23T03:14:09.4369481Z ----------------------------------------------------------------------
2022-11-23T03:14:09.4370070Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding
2022-11-23T03:14:09.4370647Z   test_sharded_embedding_colwise (__main__.TestShardedEmbedding) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:14:09.4371328Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.770s)
2022-11-23T03:14:09.4372132Z   test_sharded_embedding_rowwise (__main__.TestShardedEmbedding) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:14:09.4372529Z 
2022-11-23T03:14:09.4372781Z ----------------------------------------------------------------------
2022-11-23T03:14:09.4373126Z Ran 2 tests in 1.772s
2022-11-23T03:14:09.4373295Z 
2022-11-23T03:14:09.4373411Z OK (skipped=2)
2022-11-23T03:14:09.4373586Z 
2022-11-23T03:14:09.4373873Z Generating XML reports...
2022-11-23T03:14:09.4374562Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding/TEST-TestShardedEmbedding-20221123031407.xml
2022-11-23T03:14:09.4374937Z 
2022-11-23T03:14:09.4375415Z ##[endgroup]
2022-11-23T03:14:09.4376101Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_embedding (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_embedding_41x2jewk)
2022-11-23T03:14:09.4376518Z 
2022-11-23T03:14:09.7964683Z 
2022-11-23T03:14:09.7965376Z real	0m9.806s
2022-11-23T03:14:09.7965671Z user	0m17.291s
2022-11-23T03:14:09.7965908Z sys	0m14.375s
2022-11-23T03:14:09.7966443Z + python test/run_test.py --verbose -i distributed/_shard/sharded_tensor/ops/test_embedding_bag
2022-11-23T03:14:12.1795049Z Ignoring disabled issues:  []
2022-11-23T03:14:12.2339981Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:14:12.2340596Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:14:12.2340972Z Selected tests:
2022-11-23T03:14:12.2341285Z  distributed/_shard/sharded_tensor/ops/test_embedding_bag
2022-11-23T03:14:12.2368566Z Prioritized test from test file changes.
2022-11-23T03:14:12.2368968Z reordering tests for PR:
2022-11-23T03:14:12.2369266Z prioritized: []
2022-11-23T03:14:12.2370214Z the rest: ['distributed/_shard/sharded_tensor/ops/test_embedding_bag']
2022-11-23T03:14:12.2370369Z 
2022-11-23T03:14:12.2370925Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:14:12.2371897Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:14:12.2377505Z parallel (file granularity) tests:
2022-11-23T03:14:12.2378487Z  
2022-11-23T03:14:12.2379104Z serial (file granularity) tests:
2022-11-23T03:14:12.2379874Z  distributed/_shard/sharded_tensor/ops/test_embedding_bag
2022-11-23T03:14:14.5078801Z Ignoring disabled issues:  []
2022-11-23T03:14:14.5551995Z Ignoring disabled issues:  []
2022-11-23T03:14:14.9579632Z Running distributed/_shard/sharded_tensor/ops/test_embedding_bag ... [2022-11-23 03:14:14.957223]
2022-11-23T03:14:14.9580639Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_embedding_bag.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:14:14.957667]
2022-11-23T03:14:19.1528918Z 
2022-11-23T03:14:19.1529553Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_embedding_bag
2022-11-23T03:14:19.1530632Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_embedding_bag (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_embedding_bag_xqxh5cw2)
2022-11-23T03:14:19.1531061Z 
2022-11-23T03:14:19.1531173Z Running tests...
2022-11-23T03:14:19.1531719Z ----------------------------------------------------------------------
2022-11-23T03:14:19.1532315Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag
2022-11-23T03:14:19.1532887Z   test_sharded_embedding_bag_colwise (__main__.TestShardedEmbeddingBag) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:14:19.1533582Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.753s)
2022-11-23T03:14:19.1534400Z   test_sharded_embedding_bag_rowwise (__main__.TestShardedEmbeddingBag) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:14:19.1534797Z 
2022-11-23T03:14:19.1535078Z ----------------------------------------------------------------------
2022-11-23T03:14:19.1535392Z Ran 2 tests in 1.754s
2022-11-23T03:14:19.1535556Z 
2022-11-23T03:14:19.1535668Z OK (skipped=2)
2022-11-23T03:14:19.1535824Z 
2022-11-23T03:14:19.1535949Z Generating XML reports...
2022-11-23T03:14:19.1536603Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag/TEST-TestShardedEmbeddingBag-20221123031416.xml
2022-11-23T03:14:19.1537355Z 
2022-11-23T03:14:19.1537672Z ##[endgroup]
2022-11-23T03:14:19.1538378Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_embedding_bag (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_embedding_bag_xqxh5cw2)
2022-11-23T03:14:19.1538788Z 
2022-11-23T03:14:19.5346687Z 
2022-11-23T03:14:19.5347285Z real	0m9.738s
2022-11-23T03:14:19.5347757Z user	0m16.509s
2022-11-23T03:14:19.5348009Z sys	0m13.060s
2022-11-23T03:14:19.5348594Z + python test/run_test.py --verbose -i distributed/_shard/sharded_tensor/ops/test_binary_cmp
2022-11-23T03:14:21.9175926Z Ignoring disabled issues:  []
2022-11-23T03:14:21.9723345Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:14:21.9723936Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:14:21.9724306Z Selected tests:
2022-11-23T03:14:21.9724628Z  distributed/_shard/sharded_tensor/ops/test_binary_cmp
2022-11-23T03:14:21.9750212Z Prioritized test from test file changes.
2022-11-23T03:14:21.9750932Z reordering tests for PR:
2022-11-23T03:14:21.9751233Z prioritized: []
2022-11-23T03:14:21.9751776Z the rest: ['distributed/_shard/sharded_tensor/ops/test_binary_cmp']
2022-11-23T03:14:21.9752022Z 
2022-11-23T03:14:21.9752557Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:14:21.9753531Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:14:21.9758835Z parallel (file granularity) tests:
2022-11-23T03:14:21.9759117Z  
2022-11-23T03:14:21.9759363Z serial (file granularity) tests:
2022-11-23T03:14:21.9759708Z  distributed/_shard/sharded_tensor/ops/test_binary_cmp
2022-11-23T03:14:24.2274110Z Ignoring disabled issues:  []
2022-11-23T03:14:24.2372433Z Ignoring disabled issues:  []
2022-11-23T03:14:24.6617575Z Running distributed/_shard/sharded_tensor/ops/test_binary_cmp ... [2022-11-23 03:14:24.661067]
2022-11-23T03:14:24.6620014Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_binary_cmp.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:14:24.661532]
2022-11-23T03:14:28.8251287Z 
2022-11-23T03:14:28.8252405Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_binary_cmp
2022-11-23T03:14:28.8253467Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_binary_cmp (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_binary_cmp_w2nhzinj)
2022-11-23T03:14:28.8253915Z 
2022-11-23T03:14:28.8254040Z Running tests...
2022-11-23T03:14:28.8254600Z ----------------------------------------------------------------------
2022-11-23T03:14:28.8255208Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp
2022-11-23T03:14:28.8255629Z   test_torch_allclose (__main__.TestShardedTensorBinaryOps)
2022-11-23T03:14:28.8256082Z Test torch.allclose(ShardedTensor, ShardedTensor) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:14:28.8257108Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.753s)
2022-11-23T03:14:28.8258073Z   test_torch_allclose_tensor_specs (__main__.TestShardedTensorBinaryOps) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:14:28.8258699Z   test_torch_equal (__main__.TestShardedTensorBinaryOps)
2022-11-23T03:14:28.8259405Z Test torch.equal(ShardedTensor, ShardedTensor) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:14:28.8260270Z   test_torch_equal_tensor_specs (__main__.TestShardedTensorBinaryOps) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:14:28.8260696Z 
2022-11-23T03:14:28.8260886Z ----------------------------------------------------------------------
2022-11-23T03:14:28.8261242Z Ran 4 tests in 1.756s
2022-11-23T03:14:28.8261408Z 
2022-11-23T03:14:28.8261499Z OK (skipped=4)
2022-11-23T03:14:28.8261747Z 
2022-11-23T03:14:28.8261790Z Generating XML reports...
2022-11-23T03:14:28.8262517Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp/TEST-TestShardedTensorBinaryOps-20221123031426.xml
2022-11-23T03:14:28.8262932Z 
2022-11-23T03:14:28.8263234Z ##[endgroup]
2022-11-23T03:14:28.8263928Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_binary_cmp (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_binary_cmp_w2nhzinj)
2022-11-23T03:14:28.8264332Z 
2022-11-23T03:14:29.2021530Z 
2022-11-23T03:14:29.2022102Z real	0m9.667s
2022-11-23T03:14:29.2022452Z user	0m17.078s
2022-11-23T03:14:29.2023066Z sys	0m14.674s
2022-11-23T03:14:29.2023741Z + python test/run_test.py --verbose -i distributed/_shard/sharded_tensor/ops/test_init
2022-11-23T03:14:31.6046927Z Ignoring disabled issues:  []
2022-11-23T03:14:31.6588594Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:14:31.6589193Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:14:31.6589537Z Selected tests:
2022-11-23T03:14:31.6590040Z  distributed/_shard/sharded_tensor/ops/test_init
2022-11-23T03:14:31.6618975Z Prioritized test from test file changes.
2022-11-23T03:14:31.6619295Z reordering tests for PR:
2022-11-23T03:14:31.6619591Z prioritized: []
2022-11-23T03:14:31.6620125Z the rest: ['distributed/_shard/sharded_tensor/ops/test_init']
2022-11-23T03:14:31.6620375Z 
2022-11-23T03:14:31.6620943Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:14:31.6622218Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:14:31.6628245Z parallel (file granularity) tests:
2022-11-23T03:14:31.6628879Z  
2022-11-23T03:14:31.6629426Z serial (file granularity) tests:
2022-11-23T03:14:31.6630149Z  distributed/_shard/sharded_tensor/ops/test_init
2022-11-23T03:14:33.9361659Z Ignoring disabled issues:  []
2022-11-23T03:14:33.9404158Z Ignoring disabled issues:  []
2022-11-23T03:14:34.4136655Z Running distributed/_shard/sharded_tensor/ops/test_init ... [2022-11-23 03:14:34.413013]
2022-11-23T03:14:34.4138085Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_init.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:14:34.413451]
2022-11-23T03:14:38.5939095Z 
2022-11-23T03:14:38.5939855Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_init
2022-11-23T03:14:38.5941381Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_init (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_init_epl0k9wk)
2022-11-23T03:14:38.5941879Z 
2022-11-23T03:14:38.5942003Z Running tests...
2022-11-23T03:14:38.5942513Z ----------------------------------------------------------------------
2022-11-23T03:14:38.5943117Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init
2022-11-23T03:14:38.5943638Z   test_init_sharded_tensor_with_kaiming_uniform (__main__.TestShardedTensorNNInit)
2022-11-23T03:14:38.5944136Z Test torch.nn.init.kaiming_uniform_(ShardedTensor, a, mode, nonlinearit) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:14:38.5944828Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.751s)
2022-11-23T03:14:38.5945493Z   test_init_sharded_tensor_with_normal (__main__.TestShardedTensorNNInit)
2022-11-23T03:14:38.5946218Z Test torch.nn.init.normal_(ShardedTensor, mean, std) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:14:38.5946745Z   test_init_sharded_tensor_with_uniform (__main__.TestShardedTensorNNInit)
2022-11-23T03:14:38.5947437Z Test torch.nn.init.uniform_(ShardedTensor, a, b) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:14:38.5947799Z 
2022-11-23T03:14:38.5948063Z ----------------------------------------------------------------------
2022-11-23T03:14:38.5948396Z Ran 3 tests in 1.755s
2022-11-23T03:14:38.5948561Z 
2022-11-23T03:14:38.5948648Z OK (skipped=3)
2022-11-23T03:14:38.5948803Z 
2022-11-23T03:14:38.5948931Z Generating XML reports...
2022-11-23T03:14:38.5949773Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init/TEST-TestShardedTensorNNInit-20221123031436.xml
2022-11-23T03:14:38.5950177Z 
2022-11-23T03:14:38.5950470Z ##[endgroup]
2022-11-23T03:14:38.5951111Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_init (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_init_epl0k9wk)
2022-11-23T03:14:38.5951489Z 
2022-11-23T03:14:38.9586227Z 
2022-11-23T03:14:38.9586802Z real	0m9.756s
2022-11-23T03:14:38.9587130Z user	0m17.906s
2022-11-23T03:14:38.9587370Z sys	0m16.262s
2022-11-23T03:14:38.9587911Z + python test/run_test.py --verbose -i distributed/_shard/sharded_tensor/ops/test_linear
2022-11-23T03:14:41.3958455Z Ignoring disabled issues:  []
2022-11-23T03:14:41.4498763Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:14:41.4499388Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:14:41.4499765Z Selected tests:
2022-11-23T03:14:41.4500442Z  distributed/_shard/sharded_tensor/ops/test_linear
2022-11-23T03:14:41.4524669Z Prioritized test from test file changes.
2022-11-23T03:14:41.4525066Z reordering tests for PR:
2022-11-23T03:14:41.4525331Z prioritized: []
2022-11-23T03:14:41.4525929Z the rest: ['distributed/_shard/sharded_tensor/ops/test_linear']
2022-11-23T03:14:41.4526136Z 
2022-11-23T03:14:41.4526583Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:14:41.4527530Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:14:41.4533410Z parallel (file granularity) tests:
2022-11-23T03:14:41.4534337Z  
2022-11-23T03:14:41.4534906Z serial (file granularity) tests:
2022-11-23T03:14:41.4535523Z  distributed/_shard/sharded_tensor/ops/test_linear
2022-11-23T03:14:43.7144239Z Ignoring disabled issues:  []
2022-11-23T03:14:43.7366608Z Ignoring disabled issues:  []
2022-11-23T03:14:44.1196081Z Running distributed/_shard/sharded_tensor/ops/test_linear ... [2022-11-23 03:14:44.118975]
2022-11-23T03:14:44.1197347Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_linear.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:14:44.119409]
2022-11-23T03:14:48.2498458Z 
2022-11-23T03:14:48.2499337Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_linear
2022-11-23T03:14:48.2500400Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_linear (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_linear_mg1_veb6)
2022-11-23T03:14:48.2500833Z 
2022-11-23T03:14:48.2500954Z Running tests...
2022-11-23T03:14:48.2501573Z ----------------------------------------------------------------------
2022-11-23T03:14:48.2502177Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear
2022-11-23T03:14:48.2502630Z   test_sharded_linear_colwise (__main__.TestShardedTensorOpsLinear) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:14:48.2503474Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.706s)
2022-11-23T03:14:48.2504237Z   test_sharded_linear_errors (__main__.TestShardedTensorOpsLinear) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.003s)
2022-11-23T03:14:48.2505077Z   test_sharded_linear_rowwise (__main__.TestShardedTensorOpsLinear) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:14:48.2505486Z 
2022-11-23T03:14:48.2506062Z ----------------------------------------------------------------------
2022-11-23T03:14:48.2506403Z Ran 3 tests in 1.711s
2022-11-23T03:14:48.2506577Z 
2022-11-23T03:14:48.2506691Z OK (skipped=3)
2022-11-23T03:14:48.2506853Z 
2022-11-23T03:14:48.2506985Z Generating XML reports...
2022-11-23T03:14:48.2507648Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear/TEST-TestShardedTensorOpsLinear-20221123031446.xml
2022-11-23T03:14:48.2508048Z 
2022-11-23T03:14:48.2508358Z ##[endgroup]
2022-11-23T03:14:48.2509034Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_linear (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_linear_mg1_veb6)
2022-11-23T03:14:48.2509431Z 
2022-11-23T03:14:48.6105806Z 
2022-11-23T03:14:48.6106420Z real	0m9.652s
2022-11-23T03:14:48.6106741Z user	0m17.042s
2022-11-23T03:14:48.6106989Z sys	0m14.508s
2022-11-23T03:14:48.6107529Z + python test/run_test.py --verbose -i distributed/_shard/sharded_tensor/ops/test_math_ops
2022-11-23T03:14:50.9875790Z Ignoring disabled issues:  []
2022-11-23T03:14:51.0412427Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:14:51.0413391Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:14:51.0413958Z Selected tests:
2022-11-23T03:14:51.0414150Z  distributed/_shard/sharded_tensor/ops/test_math_ops
2022-11-23T03:14:51.0441583Z Prioritized test from test file changes.
2022-11-23T03:14:51.0441968Z reordering tests for PR:
2022-11-23T03:14:51.0442273Z prioritized: []
2022-11-23T03:14:51.0442791Z the rest: ['distributed/_shard/sharded_tensor/ops/test_math_ops']
2022-11-23T03:14:51.0443041Z 
2022-11-23T03:14:51.0443564Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:14:51.0444535Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:14:51.0450549Z parallel (file granularity) tests:
2022-11-23T03:14:51.0451226Z  
2022-11-23T03:14:51.0451532Z serial (file granularity) tests:
2022-11-23T03:14:51.0451894Z  distributed/_shard/sharded_tensor/ops/test_math_ops
2022-11-23T03:14:53.3430347Z Ignoring disabled issues:  []
2022-11-23T03:14:53.3640625Z Ignoring disabled issues:  []
2022-11-23T03:14:53.7352679Z Running distributed/_shard/sharded_tensor/ops/test_math_ops ... [2022-11-23 03:14:53.734579]
2022-11-23T03:14:53.7355361Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_math_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:14:53.735084]
2022-11-23T03:14:55.9793424Z 
2022-11-23T03:14:55.9794090Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_math_ops
2022-11-23T03:14:55.9795392Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_math_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_math_ops_csrgxduk)
2022-11-23T03:14:55.9795812Z 
2022-11-23T03:14:55.9796133Z ##[endgroup]
2022-11-23T03:14:55.9796956Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_math_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_math_ops_csrgxduk)
2022-11-23T03:14:55.9797358Z 
2022-11-23T03:14:56.3381787Z 
2022-11-23T03:14:56.3382472Z real	0m7.728s
2022-11-23T03:14:56.3382819Z user	0m16.515s
2022-11-23T03:14:56.3383074Z sys	0m15.094s
2022-11-23T03:14:56.3383613Z + python test/run_test.py --verbose -i distributed/_shard/sharded_tensor/ops/test_matrix_ops
2022-11-23T03:14:58.7309398Z Ignoring disabled issues:  []
2022-11-23T03:14:58.7865168Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:14:58.7865749Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:14:58.7866120Z Selected tests:
2022-11-23T03:14:58.7866413Z  distributed/_shard/sharded_tensor/ops/test_matrix_ops
2022-11-23T03:14:58.7893116Z Prioritized test from test file changes.
2022-11-23T03:14:58.7893529Z reordering tests for PR:
2022-11-23T03:14:58.7893825Z prioritized: []
2022-11-23T03:14:58.7894371Z the rest: ['distributed/_shard/sharded_tensor/ops/test_matrix_ops']
2022-11-23T03:14:58.7894624Z 
2022-11-23T03:14:58.7895184Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:14:58.7896145Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:14:58.7898703Z parallel (file granularity) tests:
2022-11-23T03:14:58.7899148Z  
2022-11-23T03:14:58.7899420Z serial (file granularity) tests:
2022-11-23T03:14:58.7900145Z  distributed/_shard/sharded_tensor/ops/test_matrix_ops
2022-11-23T03:15:01.1622880Z Ignoring disabled issues:  []
2022-11-23T03:15:01.1628856Z Ignoring disabled issues:  []
2022-11-23T03:15:01.6040029Z Running distributed/_shard/sharded_tensor/ops/test_matrix_ops ... [2022-11-23 03:15:01.603270]
2022-11-23T03:15:01.6040940Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_matrix_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:15:01.603796]
2022-11-23T03:15:05.8185512Z 
2022-11-23T03:15:05.8186400Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_matrix_ops
2022-11-23T03:15:05.8187651Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_matrix_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_matrix_ops_yinzr0qp)
2022-11-23T03:15:05.8188152Z 
2022-11-23T03:15:05.8188242Z Running tests...
2022-11-23T03:15:05.8188847Z ----------------------------------------------------------------------
2022-11-23T03:15:05.8189452Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_matrix_ops
2022-11-23T03:15:05.8189937Z   test_sharded_tensor_contiguous (__main__.TestShardedTensorMatrixOps) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:15:05.8190603Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.741s)
2022-11-23T03:15:05.8191421Z   test_sharded_tensor_layer_norm (__main__.TestShardedTensorMatrixOps) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:15:05.8192344Z   test_sharded_tensor_layer_norm_error (__main__.TestShardedTensorMatrixOps) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:15:05.8193254Z   test_sharded_tensor_masked_fill (__main__.TestShardedTensorMatrixOps) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:15:05.8194157Z   test_sharded_tensor_masked_fill_error (__main__.TestShardedTensorMatrixOps) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:15:05.8195063Z   test_sharded_tensor_softmax (__main__.TestShardedTensorMatrixOps) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:15:05.8195949Z   test_sharded_tensor_transpose (__main__.TestShardedTensorMatrixOps) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:15:05.8197124Z   test_sharded_tensor_transpose_error (__main__.TestShardedTensorMatrixOps) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:15:05.8198058Z   test_sharded_tensor_type_as (__main__.TestShardedTensorMatrixOps) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:15:05.8198921Z   test_sharded_tensor_view (__main__.TestShardedTensorMatrixOps) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:15:05.8199937Z   test_sharded_tensor_view_error (__main__.TestShardedTensorMatrixOps) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:15:05.8200333Z 
2022-11-23T03:15:05.8200607Z ----------------------------------------------------------------------
2022-11-23T03:15:05.8200953Z Ran 11 tests in 1.754s
2022-11-23T03:15:05.8201140Z 
2022-11-23T03:15:05.8201239Z OK (skipped=11)
2022-11-23T03:15:05.8201401Z 
2022-11-23T03:15:05.8201529Z Generating XML reports...
2022-11-23T03:15:05.8202342Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_matrix_ops/TEST-TestShardedTensorMatrixOps-20221123031503.xml
2022-11-23T03:15:05.8202747Z 
2022-11-23T03:15:05.8203044Z ##[endgroup]
2022-11-23T03:15:05.8203728Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_matrix_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_matrix_ops_yinzr0qp)
2022-11-23T03:15:05.8204132Z 
2022-11-23T03:15:06.1964614Z 
2022-11-23T03:15:06.1965084Z real	0m9.858s
2022-11-23T03:15:06.1965467Z user	0m18.020s
2022-11-23T03:15:06.1965707Z sys	0m15.956s
2022-11-23T03:15:06.1966243Z + python test/run_test.py --verbose -i distributed/_shard/sharded_tensor/ops/test_softmax
2022-11-23T03:15:08.5501475Z Ignoring disabled issues:  []
2022-11-23T03:15:08.6044640Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:15:08.6045262Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:15:08.6045627Z Selected tests:
2022-11-23T03:15:08.6045951Z  distributed/_shard/sharded_tensor/ops/test_softmax
2022-11-23T03:15:08.6070530Z Prioritized test from test file changes.
2022-11-23T03:15:08.6070894Z reordering tests for PR:
2022-11-23T03:15:08.6071285Z prioritized: []
2022-11-23T03:15:08.6071890Z the rest: ['distributed/_shard/sharded_tensor/ops/test_softmax']
2022-11-23T03:15:08.6072132Z 
2022-11-23T03:15:08.6072674Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:15:08.6073632Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:15:08.6076455Z parallel (file granularity) tests:
2022-11-23T03:15:08.6076726Z  
2022-11-23T03:15:08.6076989Z serial (file granularity) tests:
2022-11-23T03:15:08.6077333Z  distributed/_shard/sharded_tensor/ops/test_softmax
2022-11-23T03:15:10.9405022Z Ignoring disabled issues:  []
2022-11-23T03:15:10.9456625Z Ignoring disabled issues:  []
2022-11-23T03:15:11.4019227Z Running distributed/_shard/sharded_tensor/ops/test_softmax ... [2022-11-23 03:15:11.401329]
2022-11-23T03:15:11.4020331Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_softmax.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:15:11.401818]
2022-11-23T03:15:15.6598727Z 
2022-11-23T03:15:15.6599433Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_softmax
2022-11-23T03:15:15.6600801Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_softmax (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_softmax_599xerpo)
2022-11-23T03:15:15.6601516Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6hl9dfeh
2022-11-23T03:15:15.6602076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6hl9dfeh/_remote_module_non_scriptable.py
2022-11-23T03:15:15.6602390Z 
2022-11-23T03:15:15.6602505Z Running tests...
2022-11-23T03:15:15.6603029Z ----------------------------------------------------------------------
2022-11-23T03:15:15.6603631Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_softmax
2022-11-23T03:15:15.6604174Z   test_sharded_softmax_basic (__main__.TestShardedSoftmax) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:15:15.6604799Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.760s)
2022-11-23T03:15:15.6605629Z   test_sharded_softmax_on_sharding_dim (__main__.TestShardedSoftmax) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:15:15.6606208Z 
2022-11-23T03:15:15.6606483Z ----------------------------------------------------------------------
2022-11-23T03:15:15.6606829Z Ran 2 tests in 1.761s
2022-11-23T03:15:15.6606997Z 
2022-11-23T03:15:15.6607089Z OK (skipped=2)
2022-11-23T03:15:15.6607245Z 
2022-11-23T03:15:15.6607383Z Generating XML reports...
2022-11-23T03:15:15.6608029Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_softmax/TEST-TestShardedSoftmax-20221123031513.xml
2022-11-23T03:15:15.6608408Z 
2022-11-23T03:15:15.6608722Z ##[endgroup]
2022-11-23T03:15:15.6609378Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_softmax (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_softmax_599xerpo)
2022-11-23T03:15:15.6609785Z 
2022-11-23T03:15:16.0145836Z 
2022-11-23T03:15:16.0146311Z real	0m9.818s
2022-11-23T03:15:16.0146663Z user	0m16.765s
2022-11-23T03:15:16.0146932Z sys	0m13.269s
2022-11-23T03:15:16.0147530Z + python test/run_test.py --verbose -i distributed/_shard/sharded_optim/test_sharded_optim
2022-11-23T03:15:18.4043707Z Ignoring disabled issues:  []
2022-11-23T03:15:18.4589387Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:15:18.4589968Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:15:18.4590313Z Selected tests:
2022-11-23T03:15:18.4590604Z  distributed/_shard/sharded_optim/test_sharded_optim
2022-11-23T03:15:18.4619698Z Prioritized test from test file changes.
2022-11-23T03:15:18.4620045Z reordering tests for PR:
2022-11-23T03:15:18.4620299Z prioritized: []
2022-11-23T03:15:18.4621079Z the rest: ['distributed/_shard/sharded_optim/test_sharded_optim']
2022-11-23T03:15:18.4621289Z 
2022-11-23T03:15:18.4621778Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:15:18.4622728Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:15:18.4630262Z parallel (file granularity) tests:
2022-11-23T03:15:18.4630536Z  
2022-11-23T03:15:18.4630793Z serial (file granularity) tests:
2022-11-23T03:15:18.4631238Z  distributed/_shard/sharded_optim/test_sharded_optim
2022-11-23T03:15:20.7374177Z Ignoring disabled issues:  []
2022-11-23T03:15:20.7516375Z Ignoring disabled issues:  []
2022-11-23T03:15:21.1843337Z Running distributed/_shard/sharded_optim/test_sharded_optim ... [2022-11-23 03:15:21.183713]
2022-11-23T03:15:21.1844959Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_optim/test_sharded_optim.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:15:21.184160]
2022-11-23T03:19:06.4847533Z 
2022-11-23T03:19:06.4848550Z Expand the folded group to see the log file of distributed/_shard/sharded_optim/test_sharded_optim
2022-11-23T03:19:06.4849844Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_optim/test_sharded_optim (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_optim-test_sharded_optim_adsbmnio)
2022-11-23T03:19:06.4850263Z 
2022-11-23T03:19:06.4850391Z Running tests...
2022-11-23T03:19:06.4851044Z ----------------------------------------------------------------------
2022-11-23T03:19:06.4851758Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim
2022-11-23T03:19:06.4855928Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:19:06.4856752Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29477
2022-11-23T03:19:06.4857523Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29478
2022-11-23T03:19:06.4858007Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29479
2022-11-23T03:19:06.4858773Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29480
2022-11-23T03:19:06.4859472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.4859935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.4860547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.4860983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.4861553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.4861998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.4862622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.4863087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.4863679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.4864239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.4864723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.4865247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.4865749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.4866212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.4866777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.4867255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.4867812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.4868181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.4868664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.4869198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.4869645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.4870205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.4870713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.4871238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.4872019Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.4872666Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.4873284Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.4874061Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.4874368Z ok (6.144s)
2022-11-23T03:19:06.4874918Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 50
2022-11-23T03:19:06.4875469Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29766
2022-11-23T03:19:06.4875989Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29767
2022-11-23T03:19:06.4876552Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29768
2022-11-23T03:19:06.4876886Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29769
2022-11-23T03:19:06.4877582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.4877952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.4878534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.4879084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.4879582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.4880115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.4880613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.4881083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.4881734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.4882211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.4882739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.4883137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.4883812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.4884181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.4884764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.4885282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.4885732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.4886219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.4886695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.4887245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.4887662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.4888221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.4888689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.4889190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.4889872Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.4890566Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.4891234Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.4892581Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.4892996Z ok (4.317s)
2022-11-23T03:19:06.4893570Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 49
2022-11-23T03:19:06.4894200Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30055
2022-11-23T03:19:06.4894620Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30056
2022-11-23T03:19:06.4895075Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30057
2022-11-23T03:19:06.4895525Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30058
2022-11-23T03:19:06.4896116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.4896574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.4897540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.4898040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.4898591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.4898953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.4899530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.4899978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.4900558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.4901005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.4901575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.4902018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.4902594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.4903040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.4903618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.4904066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.4904507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.4904981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.4905434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.4905919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.4906504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.4907007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.4907477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.4907961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.4908727Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.4909319Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.4909984Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.4910668Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.4911068Z ok (4.317s)
2022-11-23T03:19:06.4911616Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 48
2022-11-23T03:19:06.4912225Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30344
2022-11-23T03:19:06.4912676Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30345
2022-11-23T03:19:06.4913132Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30346
2022-11-23T03:19:06.4913558Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30347
2022-11-23T03:19:06.4914169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.4914626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.4915188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.4915664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.4916280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.4916773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.4917255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.4917793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.4918375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.4918817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.4919439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.4919845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.4920412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.4920853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.4921421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.4921889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.4922308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.4922778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.4923245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.4923753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.4924261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.4924740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.4925226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.4925718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.4926362Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.4927049Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.4927730Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.4928430Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.4928853Z ok (4.417s)
2022-11-23T03:19:06.4929414Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 47
2022-11-23T03:19:06.4929948Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30633
2022-11-23T03:19:06.4930398Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30634
2022-11-23T03:19:06.4930824Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30635
2022-11-23T03:19:06.4931256Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30636
2022-11-23T03:19:06.4931861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.4932294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.4932868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.4933338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.4933911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.4934336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.4934905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.4935366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.4935925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.4936365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.4937284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.4937753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.4938255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.4938664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.4939236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.4939701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.4940170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.4940649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.4941194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.4941680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.4942165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.4942646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.4943132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.4943601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.4944258Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.4944949Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.4945633Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.4946382Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.4946763Z ok (4.418s)
2022-11-23T03:19:06.4947317Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 46
2022-11-23T03:19:06.4947866Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30922
2022-11-23T03:19:06.4948333Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30923
2022-11-23T03:19:06.4948795Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30924
2022-11-23T03:19:06.4949226Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30925
2022-11-23T03:19:06.4949858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.4950326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.4950894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.4951378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.4951971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.4952431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.4952991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.4953469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.4954065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.4954526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.4955086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.4955568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.4956171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.4956590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.4957166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.4957646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.4958101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.4958617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.4959168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.4959581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.4960049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.4960559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.4961064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.4961557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.4962202Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.4962905Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.4963605Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.4964370Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.4964743Z ok (4.318s)
2022-11-23T03:19:06.4965324Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 45
2022-11-23T03:19:06.4965869Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31211
2022-11-23T03:19:06.4966332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31212
2022-11-23T03:19:06.4966763Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31213
2022-11-23T03:19:06.4967214Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31214
2022-11-23T03:19:06.4967846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.4968296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.4968876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.4969361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.4969953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.4970385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.4970967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.4971440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.4972007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.4972461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.4973119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.4973515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.4974076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.4974532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.4975162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.4975586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.4976063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.4976555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.4977383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.4977831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.4978313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.4978715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.4979205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.4979671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.4980333Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.4981027Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.4981807Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.4982478Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.4982880Z ok (4.318s)
2022-11-23T03:19:06.4983454Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 44
2022-11-23T03:19:06.4983975Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31500
2022-11-23T03:19:06.4984437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31501
2022-11-23T03:19:06.4984896Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31502
2022-11-23T03:19:06.4985405Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31503
2022-11-23T03:19:06.4986013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.4986481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.4987065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.4987557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.4988112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.4988569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.4989150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.4989605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.4990199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.4990652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.4991238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.4991689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.4992279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.4992729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.4993286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.4993840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.4994297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.4994792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.4995265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.4995754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.4996247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.4996719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.4997215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.4997711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.4998376Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.4999111Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.4999809Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5000497Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5000885Z ok (4.318s)
2022-11-23T03:19:06.5001551Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 43
2022-11-23T03:19:06.5001972Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31789
2022-11-23T03:19:06.5002429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31790
2022-11-23T03:19:06.5002860Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31791
2022-11-23T03:19:06.5003307Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31792
2022-11-23T03:19:06.5003913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5004367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5004924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5005398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5005982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5006430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5006990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5007460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5008041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5008465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5009104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5009510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5010087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5010511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5011156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5011631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5012058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5012535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5012999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5013467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5013940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5014435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5014922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5015408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5016104Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5017089Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5017680Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5018340Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5018726Z ok (4.418s)
2022-11-23T03:19:06.5019291Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 42
2022-11-23T03:19:06.5019828Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32078
2022-11-23T03:19:06.5020260Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32079
2022-11-23T03:19:06.5020701Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32080
2022-11-23T03:19:06.5021145Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32081
2022-11-23T03:19:06.5021757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5022188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5022763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5023236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5023798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5024251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5024827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5025292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5025855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5026301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5026872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5027318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5027893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5028465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5029047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5029496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5029935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5030411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5030899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5031376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5031854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5032324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5032784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5033348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5034010Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5034700Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5035366Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5036048Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5036440Z ok (4.318s)
2022-11-23T03:19:06.5037009Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 41
2022-11-23T03:19:06.5037522Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32367
2022-11-23T03:19:06.5037978Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32368
2022-11-23T03:19:06.5038425Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32369
2022-11-23T03:19:06.5038850Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32370
2022-11-23T03:19:06.5039456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5039910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5040549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5041029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5041598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5042049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5042612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5043055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5043609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5044074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5044660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5045104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5045734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5046190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5046767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5047214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5047650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5048122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5048573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5049037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5049515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5050007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5050535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5051013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5051665Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5052351Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5053011Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5053688Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5054339Z ok (4.317s)
2022-11-23T03:19:06.5054766Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 40
2022-11-23T03:19:06.5055300Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32656
2022-11-23T03:19:06.5055825Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32657
2022-11-23T03:19:06.5056191Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32658
2022-11-23T03:19:06.5056615Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32659
2022-11-23T03:19:06.5057562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5058018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5058506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5058960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5059550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5059986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5060541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5061008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5061580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5062119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5062573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5063107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5063695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5064136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5064686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5065145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5065578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5066034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5066519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5067017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5067487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5068048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5068639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5069007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5069646Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5070329Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5071016Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5071679Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5072068Z ok (4.318s)
2022-11-23T03:19:06.5072630Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 39
2022-11-23T03:19:06.5073166Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32945
2022-11-23T03:19:06.5073596Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32946
2022-11-23T03:19:06.5074036Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32947
2022-11-23T03:19:06.5074477Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32948
2022-11-23T03:19:06.5075132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5075516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5076102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5076572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5077136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5077583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5078174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5078624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5079183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5079622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5080352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5080707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5081286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5081732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5082300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5082747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5083186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5083657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5084132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5084577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5085139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5085707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5086191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5086658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5087322Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5088010Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5088678Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5089355Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5089747Z ok (4.317s)
2022-11-23T03:19:06.5090307Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 38
2022-11-23T03:19:06.5090821Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33234
2022-11-23T03:19:06.5091275Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33235
2022-11-23T03:19:06.5091716Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 33236
2022-11-23T03:19:06.5092159Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 33237
2022-11-23T03:19:06.5092755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5093206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5093783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5094234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5094884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5095252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5095831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5096282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5097037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5097567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5098144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5098617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5099199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5099639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5100185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5100649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5101095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5101573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5102026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5102582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5103050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5103523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5104006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5104491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5105135Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5105826Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5106508Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5107191Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5107558Z ok (4.317s)
2022-11-23T03:19:06.5108117Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 37
2022-11-23T03:19:06.5108651Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33523
2022-11-23T03:19:06.5109099Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33524
2022-11-23T03:19:06.5109524Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 33525
2022-11-23T03:19:06.5109958Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 33526
2022-11-23T03:19:06.5110570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5111013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5111603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5112074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5112637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5113082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5113653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5114102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5114750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5115204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5115780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5116229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5116806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5117359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5117813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5118295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5118722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5119213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5119718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5120177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5120672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5121148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5121635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5122118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5122775Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5123513Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5124155Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5124837Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5125230Z ok (4.318s)
2022-11-23T03:19:06.5125794Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 36
2022-11-23T03:19:06.5126312Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33812
2022-11-23T03:19:06.5126762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33813
2022-11-23T03:19:06.5127212Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 33814
2022-11-23T03:19:06.5127638Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 33815
2022-11-23T03:19:06.5128261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5128712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5129285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5129739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5130316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5130762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5131335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5131840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5132427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5132877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5133429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5133894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5134470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5134987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5135564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5135941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5136380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5137338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5137853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5138328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5138706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5139170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5139658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5140194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5140871Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5141546Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5142231Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5142920Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5143311Z ok (4.317s)
2022-11-23T03:19:06.5143852Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 35
2022-11-23T03:19:06.5144381Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34101
2022-11-23T03:19:06.5144836Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34102
2022-11-23T03:19:06.5145270Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 34103
2022-11-23T03:19:06.5145713Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 34104
2022-11-23T03:19:06.5146315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5146765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5147325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5147791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5148365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5148799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5149515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5150002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5150566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5151017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5151581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5152042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5152622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5153074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5153655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5154120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5154634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5155206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5155553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5156021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5156491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5156982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5157483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5157976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5158619Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5159308Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5159999Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5160675Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5161042Z ok (4.317s)
2022-11-23T03:19:06.5161605Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 34
2022-11-23T03:19:06.5162190Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34390
2022-11-23T03:19:06.5162629Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34391
2022-11-23T03:19:06.5163078Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 34392
2022-11-23T03:19:06.5163519Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 34393
2022-11-23T03:19:06.5164136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5164569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5165146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5165616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5166201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5166687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5167270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5167739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5168303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5168745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5169350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5169784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5170342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5170792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5171362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5171860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5172302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5172775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5173263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5173690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5174169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5174663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5175154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5175628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5176285Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5177293Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5177961Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5178641Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5179036Z ok (4.317s)
2022-11-23T03:19:06.5179603Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 33
2022-11-23T03:19:06.5180114Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34679
2022-11-23T03:19:06.5180568Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34680
2022-11-23T03:19:06.5181019Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 34681
2022-11-23T03:19:06.5181461Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 34682
2022-11-23T03:19:06.5182058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5182511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5183087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5183541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5184195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5184654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5185232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5185731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5186312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5186763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5187317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5187783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5188365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5188815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5189441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5189908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5190344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5190815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5191264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5191727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5192213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5192696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5193187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5193675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5194332Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5194998Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5195686Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5196370Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5196755Z ok (4.217s)
2022-11-23T03:19:06.5197305Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 32
2022-11-23T03:19:06.5197837Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34968
2022-11-23T03:19:06.5198293Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34969
2022-11-23T03:19:06.5198727Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 34970
2022-11-23T03:19:06.5199184Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 34971
2022-11-23T03:19:06.5199800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5200253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5200810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5201331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5201918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5202494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5202944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5203410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5203988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5204414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5204981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5205445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5206017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5206507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5207082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5207541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5207964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5208458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5208940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5209410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5209876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5210344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5210838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5211334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5211987Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5212689Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5213391Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5214059Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5214462Z ok (4.317s)
2022-11-23T03:19:06.5215033Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 31
2022-11-23T03:19:06.5215580Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35257
2022-11-23T03:19:06.5216017Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35258
2022-11-23T03:19:06.5216466Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 35259
2022-11-23T03:19:06.5217264Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 35260
2022-11-23T03:19:06.5218033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5218498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5219154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5219557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5220131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5220594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5221179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5221658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5222226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5222683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5223266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5223718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5224378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5224834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5225410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5225862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5226313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5226802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5227282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5227739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5228235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5228738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5229212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5229707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5230367Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5231067Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5231744Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5232437Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5232841Z ok (4.317s)
2022-11-23T03:19:06.5233420Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 30
2022-11-23T03:19:06.5233942Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35546
2022-11-23T03:19:06.5234412Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35547
2022-11-23T03:19:06.5234876Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 35548
2022-11-23T03:19:06.5235310Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 35549
2022-11-23T03:19:06.5235930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5236443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5237039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5237497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5238092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5238551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5239107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5239583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5240230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5240698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5241263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5241797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5242392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5242953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5243411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5243886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5244342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5244824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5245327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5245809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5246288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5246762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5247262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5247758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5248405Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5249120Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5249819Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5250522Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5250896Z ok (4.317s)
2022-11-23T03:19:06.5251472Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 29
2022-11-23T03:19:06.5252016Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35835
2022-11-23T03:19:06.5252478Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35836
2022-11-23T03:19:06.5252917Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 35837
2022-11-23T03:19:06.5253366Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 35838
2022-11-23T03:19:06.5254037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5254488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5255091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5255577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5256167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5256599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5257556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5258009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5258499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5258953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5259633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5260114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5260683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5261139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5261724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5262202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5262627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5263216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5263616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5264072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5264568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5265072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5265570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5266043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5266710Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5267413Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5268115Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5268787Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5269188Z ok (4.317s)
2022-11-23T03:19:06.5269763Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 28
2022-11-23T03:19:06.5270313Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36124
2022-11-23T03:19:06.5270748Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36125
2022-11-23T03:19:06.5271195Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 36126
2022-11-23T03:19:06.5271713Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 36127
2022-11-23T03:19:06.5272329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5272794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5273400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5273885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5274457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5274917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5275500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5275957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5276552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5277065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5277653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5278109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5278709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5279176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5279721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5280198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5280653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5281160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5281637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5282116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5282607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5283098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5283561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5284058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5284733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5285464Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5286168Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5286868Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5287271Z ok (4.417s)
2022-11-23T03:19:06.5287823Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 27
2022-11-23T03:19:06.5288365Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36413
2022-11-23T03:19:06.5288828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36414
2022-11-23T03:19:06.5289398Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 36415
2022-11-23T03:19:06.5289848Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 36416
2022-11-23T03:19:06.5290472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5290937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5291502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5291987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5292577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5293031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5293597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5294077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5294834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5295186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5295748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5296228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5297078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5297617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5322024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5322556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5323003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5323498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5323987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5324451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5324965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5325479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5325987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5326472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5327162Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5327876Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5328569Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5329236Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5329642Z ok (4.317s)
2022-11-23T03:19:06.5330220Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 26
2022-11-23T03:19:06.5330924Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36702
2022-11-23T03:19:06.5331411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36703
2022-11-23T03:19:06.5331872Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 36704
2022-11-23T03:19:06.5332342Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 36705
2022-11-23T03:19:06.5332948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5333410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5334002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5334486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5335056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5335516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5336103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5336669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5337723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5338175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5338761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5339118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5339710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5340219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5340783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5341271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5341726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5342212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5342671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5343146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5343644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5344153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5344642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5345150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5345826Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5346515Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5347213Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5347908Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5348314Z ok (4.417s)
2022-11-23T03:19:06.5348949Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 25
2022-11-23T03:19:06.5349504Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36991
2022-11-23T03:19:06.5349982Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36992
2022-11-23T03:19:06.5350443Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 36993
2022-11-23T03:19:06.5350882Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 36994
2022-11-23T03:19:06.5351506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5351971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5352535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5353019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5353616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5354143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5354707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5355184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5355769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5356198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5356887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5357263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5357859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5358293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5358881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5359357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5359872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5360333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5360809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5361312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5361782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5362278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5362791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5363299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5363950Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5364651Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5365344Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5366037Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5366413Z ok (4.317s)
2022-11-23T03:19:06.5367037Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 24
2022-11-23T03:19:06.5367591Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37280
2022-11-23T03:19:06.5368032Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37281
2022-11-23T03:19:06.5368493Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 37282
2022-11-23T03:19:06.5368948Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 37283
2022-11-23T03:19:06.5369573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5370095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5370600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5371086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5371650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5372146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5372731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5373210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5373774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5374282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5374828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5375272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5375820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5376255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5377180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5377662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5378094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5378600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5378979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5379519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5379938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5380449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5380936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5381434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5382110Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5382811Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5383477Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5384248Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5384666Z ok (4.317s)
2022-11-23T03:19:06.5385254Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 23
2022-11-23T03:19:06.5385849Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37569
2022-11-23T03:19:06.5386312Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37570
2022-11-23T03:19:06.5386766Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 37571
2022-11-23T03:19:06.5387235Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 37572
2022-11-23T03:19:06.5387834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5388289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5388867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5389403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5389987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5390435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5391005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5391454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5392037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5392486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5393043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5393505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5394090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5394536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5395089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5395555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5395996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5396489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5396956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5397431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5397915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5398390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5398872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5399358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5400010Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5400683Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5401416Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5402109Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5402504Z ok (4.317s)
2022-11-23T03:19:06.5403071Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 22
2022-11-23T03:19:06.5403580Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37858
2022-11-23T03:19:06.5404040Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37859
2022-11-23T03:19:06.5404474Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 37860
2022-11-23T03:19:06.5404917Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 37861
2022-11-23T03:19:06.5405525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5405984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5406542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5407070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5407655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5408083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5408654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5409119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5409692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5410122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5410692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5411164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5411743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5412165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5412732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5413193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5413612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5414083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5414576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5415061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5415523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5416000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5416478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5417508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5418138Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5418756Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5419529Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5420195Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5420580Z ok (4.217s)
2022-11-23T03:19:06.5421016Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 21
2022-11-23T03:19:06.5421238Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38147
2022-11-23T03:19:06.5421455Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38148
2022-11-23T03:19:06.5421672Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 38149
2022-11-23T03:19:06.5421884Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 38150
2022-11-23T03:19:06.5422245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5422491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5422877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5423069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5423435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5423610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5423985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5424174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5424536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5424693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5425069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5425261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5425629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5425803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5426172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5426360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5426593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5426821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5427054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5427277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5427499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5427745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5427983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5428220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5428622Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5429065Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5429453Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5429837Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5429937Z ok (4.318s)
2022-11-23T03:19:06.5430373Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 20
2022-11-23T03:19:06.5430583Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38436
2022-11-23T03:19:06.5430804Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38437
2022-11-23T03:19:06.5431023Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 38438
2022-11-23T03:19:06.5431242Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 38439
2022-11-23T03:19:06.5431620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5431829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5432210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5432402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5432766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5432939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5433315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5433509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5433877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5434036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5434408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5434594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5434957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5435124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5435494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5435712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5435913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5436146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5436374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5436617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5436845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5437066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5437302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5437545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5437993Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5438399Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5438789Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5439165Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5439265Z ok (4.417s)
2022-11-23T03:19:06.5439701Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 19
2022-11-23T03:19:06.5439942Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38725
2022-11-23T03:19:06.5440206Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38726
2022-11-23T03:19:06.5440431Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 38727
2022-11-23T03:19:06.5440704Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 38728
2022-11-23T03:19:06.5441073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5441231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5441616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5441807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5442172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5442337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5442717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5442898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5443261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5443436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5443791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5443973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5444343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5444507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5444881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5445064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5445300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5445518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5445721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5445949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5446180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5446423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5446657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5446939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5447338Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5447738Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5448116Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5448492Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5448594Z ok (4.317s)
2022-11-23T03:19:06.5449019Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 18
2022-11-23T03:19:06.5449238Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39014
2022-11-23T03:19:06.5449460Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39015
2022-11-23T03:19:06.5449728Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 39016
2022-11-23T03:19:06.5449942Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 39017
2022-11-23T03:19:06.5450318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5450484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5450846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5451037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5451388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5451564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5451932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5452126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5452479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5452643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5452991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5453181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5453540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5453712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5454082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5454267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5454495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5454719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5454932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5455141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5455496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5455656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5455972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5456153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5456549Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5457289Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5457665Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5458072Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5458167Z ok (4.317s)
2022-11-23T03:19:06.5458581Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 17
2022-11-23T03:19:06.5458816Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39303
2022-11-23T03:19:06.5459086Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39304
2022-11-23T03:19:06.5459209Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 39305
2022-11-23T03:19:06.5459420Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 39306
2022-11-23T03:19:06.5459890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5459964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5460326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5460526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5460952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5461052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5461414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5461589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5461973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5462201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5462582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5462753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5463112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5463284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5463652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5463841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5464130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5464290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5464507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5464736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5464966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5465273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5465509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5465740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5466141Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5466527Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5466913Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5467293Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5467375Z ok (4.417s)
2022-11-23T03:19:06.5467810Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 16
2022-11-23T03:19:06.5468112Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39592
2022-11-23T03:19:06.5468319Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39593
2022-11-23T03:19:06.5468533Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 39594
2022-11-23T03:19:06.5468740Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 39595
2022-11-23T03:19:06.5469112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5469287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5469655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5469830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5470262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5470534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5470799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5470992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5471344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5471518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5471896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5472068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5472423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5472598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5472963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5473149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5473382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5473615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5473845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5474074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5474413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5474557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5474893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5475034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5475436Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5475823Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5476211Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5476601Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5476702Z ok (4.318s)
2022-11-23T03:19:06.5477119Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 15
2022-11-23T03:19:06.5477381Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39881
2022-11-23T03:19:06.5477600Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39882
2022-11-23T03:19:06.5477817Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 39883
2022-11-23T03:19:06.5478021Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 39884
2022-11-23T03:19:06.5478395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5478566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5478944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5479120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5479580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5479662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5480029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5480219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5480581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5480747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5481110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5481289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5481642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5481815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5482175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5482361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5482584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5482811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5483053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5483317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5483544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5483769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5483995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5484233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5484627Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5485021Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5485410Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5485855Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5486057Z ok (4.318s)
2022-11-23T03:19:06.5486479Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 14
2022-11-23T03:19:06.5486704Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40170
2022-11-23T03:19:06.5486925Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40171
2022-11-23T03:19:06.5487141Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 40172
2022-11-23T03:19:06.5487352Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 40173
2022-11-23T03:19:06.5487726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5487905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5488287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5488485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5488833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5489005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5489474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5489763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5490026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5490263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5490573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5490764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5491115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5491286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5491653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5491841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5492069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5492297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5492569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5492802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5493052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5493277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5493518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5493754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5494158Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5494557Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5494951Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5495393Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5495608Z ok (4.318s)
2022-11-23T03:19:06.5495929Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 13
2022-11-23T03:19:06.5496183Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40459
2022-11-23T03:19:06.5496354Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40460
2022-11-23T03:19:06.5496568Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 40461
2022-11-23T03:19:06.5497029Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 40462
2022-11-23T03:19:06.5497521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5497698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5498095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5498270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5498656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5498810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5499080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5499273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5499632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5499809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5500184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5500373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5500744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5500899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5501266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5501452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5501684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5501985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5502220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5502444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5502692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5502935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5503156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5503500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5503909Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5504197Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5504666Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5505062Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5505163Z ok (4.319s)
2022-11-23T03:19:06.5505597Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 12
2022-11-23T03:19:06.5505819Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40748
2022-11-23T03:19:06.5506020Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40749
2022-11-23T03:19:06.5506238Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 40750
2022-11-23T03:19:06.5506455Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 40751
2022-11-23T03:19:06.5506831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5507015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5507400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5507595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5507964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5508120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5508494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5508686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5509051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5509226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5509597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5509788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5510149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5510325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5510687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5510878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5511156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5511390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5511615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5511860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5512083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5512322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5512544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5512784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5513191Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5513592Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5514035Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5514432Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5514534Z ok (4.219s)
2022-11-23T03:19:06.5514967Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 11
2022-11-23T03:19:06.5515188Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41037
2022-11-23T03:19:06.5515389Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41038
2022-11-23T03:19:06.5515612Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 41039
2022-11-23T03:19:06.5515827Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 41040
2022-11-23T03:19:06.5516201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5516379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5516760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5516950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5517319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5517493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5517932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5518125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5518582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5518833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5519131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5519317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5519680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5519852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5520203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5520436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5520682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5520929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5521159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5521383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5521611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5521854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5522096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5522324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5522728Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5523177Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5523567Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5523961Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5524062Z ok (4.317s)
2022-11-23T03:19:06.5524497Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 10
2022-11-23T03:19:06.5524718Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41326
2022-11-23T03:19:06.5524941Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41327
2022-11-23T03:19:06.5525141Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 41328
2022-11-23T03:19:06.5525358Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 41329
2022-11-23T03:19:06.5525732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5525908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5526287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5526480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5526846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5527018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5527377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5527575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5527938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5528112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5528482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5528672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5529038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5529212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5529683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5529804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5530030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5530275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5530506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5530727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5530965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5531207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5531436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5531662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5532113Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5532512Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5532910Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5533298Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5533399Z ok (4.518s)
2022-11-23T03:19:06.5533831Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 9
2022-11-23T03:19:06.5534055Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41615
2022-11-23T03:19:06.5534278Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41616
2022-11-23T03:19:06.5534478Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 41617
2022-11-23T03:19:06.5534691Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 41618
2022-11-23T03:19:06.5535068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5535246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5535612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5535786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5536167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5536358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5537288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5537577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5537957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5538130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5538497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5538673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5539053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5539202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5539587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5539760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5539992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5540272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5540520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5540773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5541006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5541241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5541487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5541799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5542186Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5542589Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5542982Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5543381Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5543488Z ok (4.417s)
2022-11-23T03:19:06.5543926Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 8
2022-11-23T03:19:06.5544158Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41904
2022-11-23T03:19:06.5544383Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41905
2022-11-23T03:19:06.5544604Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 41906
2022-11-23T03:19:06.5544800Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 41907
2022-11-23T03:19:06.5545180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5545479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5545749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5546059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5546324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5546508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5546887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5547058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5547425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5547603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5547983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5548178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5548597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5548789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5549166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5549361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5549572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5549812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5550062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5550398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5550533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5550873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5551083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5551405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5551717Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5552562Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5553153Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5553864Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5554255Z ok (4.317s)
2022-11-23T03:19:06.5554835Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 7
2022-11-23T03:19:06.5555386Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42193
2022-11-23T03:19:06.5555861Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42194
2022-11-23T03:19:06.5556301Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 42195
2022-11-23T03:19:06.5556819Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 42196
2022-11-23T03:19:06.5557405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5557881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5558457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5558946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5559554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5559996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5560586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5561064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5561668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5562111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5562684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5563204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5563796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5564219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5564788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5565328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5565823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5566296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5566773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5567270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5567739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5568300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5568822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5569354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5572984Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5573722Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5574387Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5575104Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5575497Z ok (4.317s)
2022-11-23T03:19:06.5576086Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 6
2022-11-23T03:19:06.5576642Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42482
2022-11-23T03:19:06.5577612Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42483
2022-11-23T03:19:06.5578081Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 42484
2022-11-23T03:19:06.5578545Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 42485
2022-11-23T03:19:06.5579088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5579561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5580242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5580684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5581283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5581716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5582333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5582860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5583448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5583901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5585017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5585869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5586509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5586962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5587523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5587991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5588438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5588903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5589379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5589885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5590609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5591036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5591561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5592089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5592764Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5593447Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5594187Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5594890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5595290Z ok (4.317s)
2022-11-23T03:19:06.5596008Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 5
2022-11-23T03:19:06.5596517Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42771
2022-11-23T03:19:06.5596977Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42772
2022-11-23T03:19:06.5597415Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 42773
2022-11-23T03:19:06.5597867Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 42774
2022-11-23T03:19:06.5598505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5598970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5599573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5600141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5600784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5601244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5601933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5602611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5603178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5603717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5604373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5604853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5605465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5605927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5606522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5606967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5607516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5607984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5608370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5609538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5610011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5610482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5611039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5611447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5612113Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5612806Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5613489Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5614151Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5614534Z ok (4.317s)
2022-11-23T03:19:06.5615090Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 4
2022-11-23T03:19:06.5615622Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43060
2022-11-23T03:19:06.5616063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43061
2022-11-23T03:19:06.5616505Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 43062
2022-11-23T03:19:06.5617396Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 43063
2022-11-23T03:19:06.5618016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5618474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5619029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5619414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5619976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5620423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5620992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5621457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5622098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5622560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5623135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5623583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5624161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5624602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5625170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5625757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5626277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5626681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5627229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5627745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5628212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5628690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5629279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5629856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5630553Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5631242Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5631916Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5632598Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5632990Z ok (4.417s)
2022-11-23T03:19:06.5633528Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 3
2022-11-23T03:19:06.5634053Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43349
2022-11-23T03:19:06.5634509Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43350
2022-11-23T03:19:06.5634964Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 43351
2022-11-23T03:19:06.5635394Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 43352
2022-11-23T03:19:06.5636005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5636459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5637032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5637581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5638241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5638592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5639147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5639666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5640308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5640755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5641316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5641779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5642356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5642786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5643358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5643834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5644462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5645365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5645964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5646452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5646924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5647383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5647862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5648358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5649019Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5649716Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5650398Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5651079Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5651449Z ok (4.417s)
2022-11-23T03:19:06.5652006Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 2
2022-11-23T03:19:06.5652532Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43638
2022-11-23T03:19:06.5652991Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43639
2022-11-23T03:19:06.5653426Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 43640
2022-11-23T03:19:06.5653871Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 43641
2022-11-23T03:19:06.5654558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5655073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5655623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5656016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5656596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5657488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5658156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5658653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5659110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5659552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5660120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5660581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5661141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5661668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5662155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5662616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5663107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5663600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5664091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5664537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5665015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5665491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5665962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5666436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5667103Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5667792Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5668470Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5669134Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5669525Z ok (4.317s)
2022-11-23T03:19:06.5670080Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 1
2022-11-23T03:19:06.5670598Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43927
2022-11-23T03:19:06.5671056Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43928
2022-11-23T03:19:06.5671510Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 43929
2022-11-23T03:19:06.5671958Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 43930
2022-11-23T03:19:06.5672551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5673000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5673573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5674023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5674605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5675103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5675784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5676130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5676734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5677147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5677717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5678191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5678851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:06.5679295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:06.5679731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:06.5680264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:06.5680690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:19:06.5681164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:19:06.5681635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T03:19:06.5682114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:19:06.5682593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T03:19:06.5683049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:19:06.5683527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T03:19:06.5684023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T03:19:06.5684685Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5685355Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5686098Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5686782Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T03:19:06.5687169Z ok (4.318s)
2022-11-23T03:19:06.5687677Z   test_sharded_optim (__main__.TestShardedOptimizer) ...     test_named_params_with_sharded_tensor succeeded - num_retries_left: 0
2022-11-23T03:19:06.5688344Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:19:06.5688652Z 
2022-11-23T03:19:06.5688920Z ----------------------------------------------------------------------
2022-11-23T03:19:06.5689248Z Ran 52 tests in 222.922s
2022-11-23T03:19:06.5689396Z 
2022-11-23T03:19:06.5689505Z OK (skipped=1)
2022-11-23T03:19:06.5689660Z 
2022-11-23T03:19:06.5689783Z Generating XML reports...
2022-11-23T03:19:06.5690424Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim/TEST-TestShardedOptimizer-20221123031523.xml
2022-11-23T03:19:06.5690807Z 
2022-11-23T03:19:06.5691372Z ##[endgroup]
2022-11-23T03:19:06.5692048Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_optim/test_sharded_optim (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_optim-test_sharded_optim_adsbmnio)
2022-11-23T03:19:06.5692551Z 
2022-11-23T03:19:06.8706137Z 
2022-11-23T03:19:06.8706820Z real	3m50.856s
2022-11-23T03:19:06.8707129Z user	13m9.510s
2022-11-23T03:19:06.8707366Z sys	8m52.091s
2022-11-23T03:19:06.8708023Z + python test/run_test.py --verbose -i distributed/_shard/test_partial_tensor
2022-11-23T03:19:09.2429825Z Ignoring disabled issues:  []
2022-11-23T03:19:09.2975345Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:19:09.2975932Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:19:09.2976312Z Selected tests:
2022-11-23T03:19:09.2976600Z  distributed/_shard/test_partial_tensor
2022-11-23T03:19:09.3003973Z Prioritized test from test file changes.
2022-11-23T03:19:09.3004677Z reordering tests for PR:
2022-11-23T03:19:09.3004997Z prioritized: []
2022-11-23T03:19:09.3005500Z the rest: ['distributed/_shard/test_partial_tensor']
2022-11-23T03:19:09.3005719Z 
2022-11-23T03:19:09.3006268Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:19:09.3007491Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:19:09.3013623Z parallel (file granularity) tests:
2022-11-23T03:19:09.3014169Z  
2022-11-23T03:19:09.3014476Z serial (file granularity) tests:
2022-11-23T03:19:09.3014797Z  distributed/_shard/test_partial_tensor
2022-11-23T03:19:11.5803374Z Ignoring disabled issues:  []
2022-11-23T03:19:11.6053001Z Ignoring disabled issues:  []
2022-11-23T03:19:11.9773633Z Running distributed/_shard/test_partial_tensor ... [2022-11-23 03:19:11.976737]
2022-11-23T03:19:11.9774588Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/test_partial_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:19:11.977175]
2022-11-23T03:19:16.1571745Z 
2022-11-23T03:19:16.1572479Z Expand the folded group to see the log file of distributed/_shard/test_partial_tensor
2022-11-23T03:19:16.1574048Z ##[group]PRINTING LOG FILE of distributed/_shard/test_partial_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_partial_tensor_v_xrgz3z)
2022-11-23T03:19:16.1574448Z 
2022-11-23T03:19:16.1574573Z Running tests...
2022-11-23T03:19:16.1575161Z ----------------------------------------------------------------------
2022-11-23T03:19:16.1575785Z Test results will be stored in test-reports/python-unittest/distributed._shard.test_partial_tensor
2022-11-23T03:19:16.1577073Z   test_cat (__main__.TestPartialTensorOps) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:19:16.1578207Z skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (1.767s)
2022-11-23T03:19:16.1579775Z   test_cat_errors (__main__.TestPartialTensorOps) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:19:16.1580640Z   test_transpose (__main__.TestPartialTensorOps) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:19:16.1581515Z   test_partial_tensor_reshard (__main__.TestPartialTensorReshard) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.001s)
2022-11-23T03:19:16.1582420Z   test_partial_tensor_reshard_errors (__main__.TestPartialTensorReshard) ... skip: Test is enabled but --rerun-disabled-tests verification mode is set, so only disabled tests are run (0.002s)
2022-11-23T03:19:16.1582831Z 
2022-11-23T03:19:16.1583110Z ----------------------------------------------------------------------
2022-11-23T03:19:16.1583486Z Ran 5 tests in 1.772s
2022-11-23T03:19:16.1583730Z 
2022-11-23T03:19:16.1583774Z OK (skipped=5)
2022-11-23T03:19:16.1584195Z 
2022-11-23T03:19:16.1584345Z Generating XML reports...
2022-11-23T03:19:16.1585001Z Generated XML report: test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorOps-20221123031913.xml
2022-11-23T03:19:16.1585835Z Generated XML report: test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorReshard-20221123031913.xml
2022-11-23T03:19:16.1586218Z 
2022-11-23T03:19:16.1586551Z ##[endgroup]
2022-11-23T03:19:16.1587180Z FINISHED PRINTING LOG FILE of distributed/_shard/test_partial_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_partial_tensor_v_xrgz3z)
2022-11-23T03:19:16.1587555Z 
2022-11-23T03:19:16.5139840Z 
2022-11-23T03:19:16.5140576Z real	0m9.643s
2022-11-23T03:19:16.5141253Z user	0m17.616s
2022-11-23T03:19:16.5141556Z sys	0m15.822s
2022-11-23T03:19:16.5142197Z + python test/run_test.py --verbose -i distributed/_shard/test_replicated_tensor
2022-11-23T03:19:18.9080664Z Ignoring disabled issues:  []
2022-11-23T03:19:18.9629910Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:19:18.9630808Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:19:18.9631184Z Selected tests:
2022-11-23T03:19:18.9631471Z  distributed/_shard/test_replicated_tensor
2022-11-23T03:19:18.9656481Z Prioritized test from test file changes.
2022-11-23T03:19:18.9657528Z reordering tests for PR:
2022-11-23T03:19:18.9657847Z prioritized: []
2022-11-23T03:19:18.9658409Z the rest: ['distributed/_shard/test_replicated_tensor']
2022-11-23T03:19:18.9658630Z 
2022-11-23T03:19:18.9659170Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:19:18.9660060Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:19:18.9665277Z parallel (file granularity) tests:
2022-11-23T03:19:18.9665580Z  
2022-11-23T03:19:18.9665953Z serial (file granularity) tests:
2022-11-23T03:19:18.9666247Z  distributed/_shard/test_replicated_tensor
2022-11-23T03:19:21.3111189Z Ignoring disabled issues:  []
2022-11-23T03:19:21.3134612Z Ignoring disabled issues:  []
2022-11-23T03:19:21.7102526Z Running distributed/_shard/test_replicated_tensor ... [2022-11-23 03:19:21.709720]
2022-11-23T03:19:21.7105731Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/test_replicated_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:19:21.710182]
2022-11-23T03:19:24.0246450Z 
2022-11-23T03:19:24.0247105Z Expand the folded group to see the log file of distributed/_shard/test_replicated_tensor
2022-11-23T03:19:24.0248281Z ##[group]PRINTING LOG FILE of distributed/_shard/test_replicated_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_replicated_tensor_7rqrwhvx)
2022-11-23T03:19:24.0248680Z 
2022-11-23T03:19:24.0248990Z ##[endgroup]
2022-11-23T03:19:24.0249764Z FINISHED PRINTING LOG FILE of distributed/_shard/test_replicated_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_replicated_tensor_7rqrwhvx)
2022-11-23T03:19:24.0250145Z 
2022-11-23T03:19:24.3894048Z 
2022-11-23T03:19:24.3894505Z real	0m7.875s
2022-11-23T03:19:24.3894882Z user	0m17.053s
2022-11-23T03:19:24.3895142Z sys	0m15.421s
2022-11-23T03:19:24.3895705Z + python test/run_test.py --verbose -i test_cuda_primary_ctx
2022-11-23T03:19:26.8057944Z Ignoring disabled issues:  []
2022-11-23T03:19:26.8604903Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T03:19:26.8605486Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T03:19:26.8606151Z Selected tests:
2022-11-23T03:19:26.8606429Z  test_cuda_primary_ctx
2022-11-23T03:19:26.8634115Z Prioritized test from test file changes.
2022-11-23T03:19:26.8634691Z reordering tests for PR:
2022-11-23T03:19:26.8635022Z prioritized: []
2022-11-23T03:19:26.8635459Z the rest: ['test_cuda_primary_ctx']
2022-11-23T03:19:26.8635654Z 
2022-11-23T03:19:26.8636197Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T03:19:26.8637152Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T03:19:26.8644461Z parallel (file granularity) tests:
2022-11-23T03:19:26.8645054Z  
2022-11-23T03:19:26.8645576Z serial (file granularity) tests:
2022-11-23T03:19:26.8645919Z  test_cuda_primary_ctx
2022-11-23T03:19:29.2725281Z Ignoring disabled issues:  []
2022-11-23T03:19:29.2891600Z Ignoring disabled issues:  []
2022-11-23T03:19:29.7351074Z Running test_cuda_primary_ctx ... [2022-11-23 03:19:29.734467]
2022-11-23T03:19:29.7352573Z Executing ['/opt/conda/bin/python', '-bb', 'test_cuda_primary_ctx.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests', '--rerun-disabled-tests'] ... [2022-11-23 03:19:29.734919]
2022-11-23T03:19:46.7532571Z 
2022-11-23T03:19:46.7533426Z Expand the folded group to see the log file of test_cuda_primary_ctx
2022-11-23T03:19:46.7534409Z ##[group]PRINTING LOG FILE of test_cuda_primary_ctx (/var/lib/jenkins/workspace/test/test-reports/test_cuda_primary_ctx_xcunt0lu)
2022-11-23T03:19:46.7534762Z <unittest.suite.TestSuite tests=[]>
2022-11-23T03:19:46.7535385Z <unittest.suite.TestSuite tests=[<__main__.TestCudaPrimaryCtx testMethod=test_copy>, <__main__.TestCudaPrimaryCtx testMethod=test_pin_memory>, <__main__.TestCudaPrimaryCtx testMethod=test_str_repr>]>
2022-11-23T03:19:46.7536031Z test_copy (__main__.TestCudaPrimaryCtx)
2022-11-23T03:19:46.7536381Z test_pin_memory (__main__.TestCudaPrimaryCtx)
2022-11-23T03:19:46.7536686Z test_str_repr (__main__.TestCudaPrimaryCtx)
2022-11-23T03:19:46.7538010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:46.7538469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:46.7539088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:46.7539577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:46.7539887Z 
2022-11-23T03:19:46.7540010Z Running tests...
2022-11-23T03:19:46.7540337Z ----------------------------------------------------------------------
2022-11-23T03:19:46.7540995Z Test results will be stored in test-reports/python-unittest/test_cuda_primary_ctx
2022-11-23T03:19:46.7541471Z   test_copy (__main__.TestCudaPrimaryCtx) ... ok (1.400s)
2022-11-23T03:19:46.7541660Z 
2022-11-23T03:19:46.7541944Z ----------------------------------------------------------------------
2022-11-23T03:19:46.7542245Z Ran 1 test in 2.516s
2022-11-23T03:19:46.7542370Z 
2022-11-23T03:19:46.7542460Z OK
2022-11-23T03:19:46.7542683Z 
2022-11-23T03:19:46.7542735Z Generating XML reports...
2022-11-23T03:19:46.7543395Z Generated XML report: test-reports/python-unittest/test_cuda_primary_ctx/TEST-TestCudaPrimaryCtx-20221123031933.xml
2022-11-23T03:19:46.7544006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:46.7544574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:46.7545066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:46.7545517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:46.7545756Z 
2022-11-23T03:19:46.7545928Z Running tests...
2022-11-23T03:19:46.7546604Z ----------------------------------------------------------------------
2022-11-23T03:19:46.7547163Z Test results will be stored in test-reports/python-unittest/test_cuda_primary_ctx
2022-11-23T03:19:46.7547571Z   test_pin_memory (__main__.TestCudaPrimaryCtx) ... ok (1.420s)
2022-11-23T03:19:46.7547902Z 
2022-11-23T03:19:46.7548182Z ----------------------------------------------------------------------
2022-11-23T03:19:46.7548522Z Ran 1 test in 2.510s
2022-11-23T03:19:46.7548689Z 
2022-11-23T03:19:46.7548792Z OK
2022-11-23T03:19:46.7548907Z 
2022-11-23T03:19:46.7549034Z Generating XML reports...
2022-11-23T03:19:46.7549621Z Generated XML report: test-reports/python-unittest/test_cuda_primary_ctx/TEST-TestCudaPrimaryCtx-20221123031938.xml
2022-11-23T03:19:46.7550321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:19:46.7550761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:19:46.7551357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:19:46.7552024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:19:46.7552261Z 
2022-11-23T03:19:46.7552375Z Running tests...
2022-11-23T03:19:46.7552773Z ----------------------------------------------------------------------
2022-11-23T03:19:46.7553303Z Test results will be stored in test-reports/python-unittest/test_cuda_primary_ctx
2022-11-23T03:19:46.7553735Z   test_str_repr (__main__.TestCudaPrimaryCtx) ... ok (1.336s)
2022-11-23T03:19:46.7553957Z 
2022-11-23T03:19:46.7554204Z ----------------------------------------------------------------------
2022-11-23T03:19:46.7554533Z Ran 1 test in 2.392s
2022-11-23T03:19:46.7554703Z 
2022-11-23T03:19:46.7554808Z OK
2022-11-23T03:19:46.7554946Z 
2022-11-23T03:19:46.7555075Z Generating XML reports...
2022-11-23T03:19:46.7555637Z Generated XML report: test-reports/python-unittest/test_cuda_primary_ctx/TEST-TestCudaPrimaryCtx-20221123031943.xml
2022-11-23T03:19:46.7556004Z 
2022-11-23T03:19:46.7556311Z ##[endgroup]
2022-11-23T03:19:46.7556836Z FINISHED PRINTING LOG FILE of test_cuda_primary_ctx (/var/lib/jenkins/workspace/test/test-reports/test_cuda_primary_ctx_xcunt0lu)
2022-11-23T03:19:46.7557155Z 
2022-11-23T03:19:47.1080438Z 
2022-11-23T03:19:47.1080983Z real	0m22.719s
2022-11-23T03:19:47.1081265Z user	0m33.538s
2022-11-23T03:19:47.1081517Z sys	0m28.043s
2022-11-23T03:19:47.1082873Z + assert_git_not_dirty
2022-11-23T03:19:47.1083558Z + [[ linux-bionic-cuda11.6-py3.9-gcc7 != *rocm* ]]
2022-11-23T03:19:47.1083898Z + [[ linux-bionic-cuda11.6-py3.9-gcc7 != *xla* ]]
2022-11-23T03:19:47.1088070Z ++ git status --porcelain
2022-11-23T03:19:48.7049173Z + git_status=
2022-11-23T03:19:48.7049709Z + [[ -n '' ]]
2022-11-23T03:19:48.7129211Z Prepare all required actions
2022-11-23T03:19:48.7129626Z Getting action download info
2022-11-23T03:19:48.8711435Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482)
2022-11-23T03:19:49.0605742Z ##[group]Run ./.github/actions/get-workflow-job-id
2022-11-23T03:19:49.0606049Z with:
2022-11-23T03:19:49.0606601Z   github-token: ***
2022-11-23T03:19:49.0606850Z env:
2022-11-23T03:19:49.0607077Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:19:49.0607505Z   GPU_FLAG: --gpus all
2022-11-23T03:19:49.0607866Z   DOCKER_CONTAINER_ID: bd7ceb9753ed791b4138e1687453f278a68449d77e2fd64dd747f8de67c8d5c5
2022-11-23T03:19:49.0608214Z ##[endgroup]
2022-11-23T03:19:49.0641483Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2022-11-23T03:19:49.0641709Z with:
2022-11-23T03:19:49.0641934Z   shell: bash
2022-11-23T03:19:49.0642161Z   timeout_minutes: 10
2022-11-23T03:19:49.0642408Z   max_attempts: 5
2022-11-23T03:19:49.0642693Z   retry_wait_seconds: 30
2022-11-23T03:19:49.0643183Z   command: set -eux
python3 -m pip install requests==2.26.0
GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}")
echo "job-id=${GHA_WORKFLOW_JOB_ID}" >> "${GITHUB_OUTPUT}"

2022-11-23T03:19:49.0643690Z   polling_interval_seconds: 1
2022-11-23T03:19:49.0643957Z   warning_on_retry: true
2022-11-23T03:19:49.0644220Z   continue_on_error: false
2022-11-23T03:19:49.0644445Z env:
2022-11-23T03:19:49.0644682Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:19:49.0644979Z   GPU_FLAG: --gpus all
2022-11-23T03:19:49.0645288Z   DOCKER_CONTAINER_ID: bd7ceb9753ed791b4138e1687453f278a68449d77e2fd64dd747f8de67c8d5c5
2022-11-23T03:19:49.0645776Z   GITHUB_TOKEN: ***
2022-11-23T03:19:49.0646024Z ##[endgroup]
2022-11-23T03:19:49.1342386Z + python3 -m pip install requests==2.26.0
2022-11-23T03:19:49.4307788Z Defaulting to user installation because normal site-packages is not writeable
2022-11-23T03:19:49.5776243Z Collecting requests==2.26.0
2022-11-23T03:19:49.5983547Z   Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB)
2022-11-23T03:19:49.7227874Z Collecting certifi>=2017.4.17
2022-11-23T03:19:49.7269917Z   Downloading certifi-2022.9.24-py3-none-any.whl (161 kB)
2022-11-23T03:19:49.7814528Z Collecting idna<4,>=2.5; python_version >= "3"
2022-11-23T03:19:49.7866209Z   Downloading idna-3.4-py3-none-any.whl (61 kB)
2022-11-23T03:19:49.9723224Z Collecting charset-normalizer~=2.0.0; python_version >= "3"
2022-11-23T03:19:49.9780518Z   Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
2022-11-23T03:19:50.0881060Z Collecting urllib3<1.27,>=1.21.1
2022-11-23T03:19:50.0925629Z   Downloading urllib3-1.26.12-py2.py3-none-any.whl (140 kB)
2022-11-23T03:19:50.2059143Z Installing collected packages: certifi, idna, charset-normalizer, urllib3, requests
2022-11-23T03:19:50.3381808Z   WARNING: The script normalizer is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-11-23T03:19:50.3382616Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-11-23T03:19:50.4762723Z Successfully installed certifi-2022.9.24 charset-normalizer-2.0.12 idna-3.4 requests-2.26.0 urllib3-1.26.12
2022-11-23T03:19:50.5307202Z ++ python3 .github/scripts/get_workflow_job_id.py 3528394938 i-018c617fc63e71812
2022-11-23T03:19:54.3421752Z + GHA_WORKFLOW_JOB_ID=9655554784
2022-11-23T03:19:54.3422342Z + echo job-id=9655554784
2022-11-23T03:19:55.1373287Z Command completed after 1 attempt(s).
2022-11-23T03:19:55.1532819Z ##[group]Run kill "$MONITOR_SCRIPT_PID"
2022-11-23T03:19:55.1533185Z kill "$MONITOR_SCRIPT_PID"
2022-11-23T03:19:55.1549306Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T03:19:55.1549613Z env:
2022-11-23T03:19:55.1549856Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:19:55.1550103Z   GPU_FLAG: --gpus all
2022-11-23T03:19:55.1550465Z   DOCKER_CONTAINER_ID: bd7ceb9753ed791b4138e1687453f278a68449d77e2fd64dd747f8de67c8d5c5
2022-11-23T03:19:55.1550832Z   MONITOR_SCRIPT_PID: 91918
2022-11-23T03:19:55.1551073Z ##[endgroup]
2022-11-23T03:19:55.1682078Z Prepare all required actions
2022-11-23T03:19:55.1682453Z Getting action download info
2022-11-23T03:19:55.3805386Z Download action repository 'actions/upload-artifact@v3' (SHA:83fd05a356d7e2593de66fc9913b3002723633cb)
2022-11-23T03:19:55.5637674Z ##[group]Run ./.github/actions/upload-test-artifacts
2022-11-23T03:19:55.5638077Z with:
2022-11-23T03:19:55.5638364Z   file-suffix: test-multigpu-1-1-linux.16xlarge.nvidia.gpu_9655554784
2022-11-23T03:19:55.5638652Z env:
2022-11-23T03:19:55.5638874Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:19:55.5639141Z   GPU_FLAG: --gpus all
2022-11-23T03:19:55.5639505Z   DOCKER_CONTAINER_ID: bd7ceb9753ed791b4138e1687453f278a68449d77e2fd64dd747f8de67c8d5c5
2022-11-23T03:19:55.5639827Z ##[endgroup]
2022-11-23T03:19:55.5671934Z ##[group]Run # Remove any previous test jsons if they exist
2022-11-23T03:19:55.5672476Z # Remove any previous test jsons if they exist
2022-11-23T03:19:55.5672789Z rm -f test-jsons-*.zip
2022-11-23T03:19:55.5673160Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json'
2022-11-23T03:19:55.5685297Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T03:19:55.5685642Z env:
2022-11-23T03:19:55.5685840Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:19:55.5686089Z   GPU_FLAG: --gpus all
2022-11-23T03:19:55.5686447Z   DOCKER_CONTAINER_ID: bd7ceb9753ed791b4138e1687453f278a68449d77e2fd64dd747f8de67c8d5c5
2022-11-23T03:19:55.5687065Z   FILE_SUFFIX: test-multigpu-1-1-linux.16xlarge.nvidia.gpu_9655554784
2022-11-23T03:19:55.5687490Z ##[endgroup]
2022-11-23T03:19:55.5838545Z   adding: test/allowlist_for_publicAPI.json (deflated 79%)
2022-11-23T03:19:55.5874938Z   adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%)
2022-11-23T03:19:55.5881666Z   adding: test/profiler/profiler_utils_mock_events.json (deflated 87%)
2022-11-23T03:19:55.5882387Z   adding: test/.pytorch-slow-tests.json (deflated 73%)
2022-11-23T03:19:55.5894753Z   adding: test/.pytorch-disabled-tests.json (deflated 86%)
2022-11-23T03:19:55.5921545Z ##[group]Run # Remove any previous test reports if they exist
2022-11-23T03:19:55.5921945Z # Remove any previous test reports if they exist
2022-11-23T03:19:55.5922607Z rm -f test-reports-*.zip
2022-11-23T03:19:55.5922987Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv'
2022-11-23T03:19:55.5938581Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T03:19:55.5938896Z env:
2022-11-23T03:19:55.5939153Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:19:55.5939414Z   GPU_FLAG: --gpus all
2022-11-23T03:19:55.5939795Z   DOCKER_CONTAINER_ID: bd7ceb9753ed791b4138e1687453f278a68449d77e2fd64dd747f8de67c8d5c5
2022-11-23T03:19:55.5940274Z   FILE_SUFFIX: test-multigpu-1-1-linux.16xlarge.nvidia.gpu_9655554784
2022-11-23T03:19:55.5940615Z ##[endgroup]
2022-11-23T03:19:55.6079147Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20221123014921.xml (deflated 37%)
2022-11-23T03:19:55.6079964Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221123014927.xml (deflated 41%)
2022-11-23T03:19:55.6080811Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221123014931.xml (deflated 40%)
2022-11-23T03:19:55.6081639Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221123014935.xml (deflated 40%)
2022-11-23T03:19:55.6082460Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221123014939.xml (deflated 42%)
2022-11-23T03:19:55.6083302Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221123014944.xml (deflated 41%)
2022-11-23T03:19:55.6084170Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221123014950.xml (deflated 41%)
2022-11-23T03:19:55.6085310Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221123014958.xml (deflated 41%)
2022-11-23T03:19:55.6086206Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221123015005.xml (deflated 41%)
2022-11-23T03:19:55.6086981Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20221123015015.xml (deflated 39%)
2022-11-23T03:19:55.6087693Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20221123015019.xml (deflated 39%)
2022-11-23T03:19:55.6088399Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20221123015023.xml (deflated 39%)
2022-11-23T03:19:55.6089087Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123015035.xml (deflated 38%)
2022-11-23T03:19:55.6089764Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123015041.xml (deflated 38%)
2022-11-23T03:19:55.6090454Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123015050.xml (deflated 38%)
2022-11-23T03:19:55.6091136Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123015056.xml (deflated 37%)
2022-11-23T03:19:55.6091808Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123015103.xml (deflated 38%)
2022-11-23T03:19:55.6092458Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123015111.xml (deflated 38%)
2022-11-23T03:19:55.6093133Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123015119.xml (deflated 38%)
2022-11-23T03:19:55.6093802Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123015126.xml (deflated 38%)
2022-11-23T03:19:55.6094481Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123015132.xml (deflated 38%)
2022-11-23T03:19:55.6095128Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123015139.xml (deflated 37%)
2022-11-23T03:19:55.6095925Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123015146.xml (deflated 37%)
2022-11-23T03:19:55.6096617Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123015152.xml (deflated 38%)
2022-11-23T03:19:55.6097649Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123015159.xml (deflated 37%)
2022-11-23T03:19:55.6098333Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123015207.xml (deflated 38%)
2022-11-23T03:19:55.6099036Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123015214.xml (deflated 37%)
2022-11-23T03:19:55.6099746Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123015222.xml (deflated 38%)
2022-11-23T03:19:55.6100445Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123015229.xml (deflated 38%)
2022-11-23T03:19:55.6101136Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123015237.xml (deflated 39%)
2022-11-23T03:19:55.6101842Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123015244.xml (deflated 38%)
2022-11-23T03:19:55.6102548Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123015252.xml (deflated 38%)
2022-11-23T03:19:55.6103240Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123015259.xml (deflated 38%)
2022-11-23T03:19:55.6103911Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123015305.xml (deflated 38%)
2022-11-23T03:19:55.6104843Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015313.xml (deflated 44%)
2022-11-23T03:19:55.6105687Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015322.xml (deflated 45%)
2022-11-23T03:19:55.6106512Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015331.xml (deflated 43%)
2022-11-23T03:19:55.6107294Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015340.xml (deflated 43%)
2022-11-23T03:19:55.6108106Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015348.xml (deflated 45%)
2022-11-23T03:19:55.6108903Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015357.xml (deflated 45%)
2022-11-23T03:19:55.6109715Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015406.xml (deflated 46%)
2022-11-23T03:19:55.6110534Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015415.xml (deflated 47%)
2022-11-23T03:19:55.6111331Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015424.xml (deflated 44%)
2022-11-23T03:19:55.6112132Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015432.xml (deflated 46%)
2022-11-23T03:19:55.6112935Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015441.xml (deflated 45%)
2022-11-23T03:19:55.6113738Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015450.xml (deflated 44%)
2022-11-23T03:19:55.6114522Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015459.xml (deflated 44%)
2022-11-23T03:19:55.6115322Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015508.xml (deflated 43%)
2022-11-23T03:19:55.6116242Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015514.xml (deflated 44%)
2022-11-23T03:19:55.6117047Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015522.xml (deflated 45%)
2022-11-23T03:19:55.6117833Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015529.xml (deflated 44%)
2022-11-23T03:19:55.6118628Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015536.xml (deflated 45%)
2022-11-23T03:19:55.6119434Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015542.xml (deflated 45%)
2022-11-23T03:19:55.6120274Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015549.xml (deflated 50%)
2022-11-23T03:19:55.6121082Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015557.xml (deflated 42%)
2022-11-23T03:19:55.6121867Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015605.xml (deflated 41%)
2022-11-23T03:19:55.6122666Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015613.xml (deflated 41%)
2022-11-23T03:19:55.6123463Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015622.xml (deflated 41%)
2022-11-23T03:19:55.6124318Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015630.xml (deflated 42%)
2022-11-23T03:19:55.6125115Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015639.xml (deflated 41%)
2022-11-23T03:19:55.6125916Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015649.xml (deflated 42%)
2022-11-23T03:19:55.6126714Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015656.xml (deflated 41%)
2022-11-23T03:19:55.6127509Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015703.xml (deflated 41%)
2022-11-23T03:19:55.6128291Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015709.xml (deflated 44%)
2022-11-23T03:19:55.6129087Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015716.xml (deflated 45%)
2022-11-23T03:19:55.6129887Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015723.xml (deflated 41%)
2022-11-23T03:19:55.6130687Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015731.xml (deflated 41%)
2022-11-23T03:19:55.6131460Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015737.xml (deflated 41%)
2022-11-23T03:19:55.6132250Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015747.xml (deflated 41%)
2022-11-23T03:19:55.6133052Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015754.xml (deflated 41%)
2022-11-23T03:19:55.6133854Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015801.xml (deflated 41%)
2022-11-23T03:19:55.6134627Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123015810.xml (deflated 41%)
2022-11-23T03:19:55.6135595Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20221123015819.xml (deflated 42%)
2022-11-23T03:19:55.6136582Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20221123015826.xml (deflated 42%)
2022-11-23T03:19:55.6137771Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20221123015832.xml (deflated 43%)
2022-11-23T03:19:55.6138718Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20221123015839.xml (deflated 42%)
2022-11-23T03:19:55.6139637Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123015845.xml (deflated 39%)
2022-11-23T03:19:55.6140410Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123015852.xml (deflated 39%)
2022-11-23T03:19:55.6141153Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123020447.xml (deflated 39%)
2022-11-23T03:19:55.6141872Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123015901.xml (deflated 39%)
2022-11-23T03:19:55.6142636Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123015908.xml (deflated 40%)
2022-11-23T03:19:55.6143402Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123015914.xml (deflated 39%)
2022-11-23T03:19:55.6144158Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123015921.xml (deflated 39%)
2022-11-23T03:19:55.6144972Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123015928.xml (deflated 39%)
2022-11-23T03:19:55.6145753Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123015935.xml (deflated 39%)
2022-11-23T03:19:55.6146519Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123015945.xml (deflated 40%)
2022-11-23T03:19:55.6147269Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123015951.xml (deflated 39%)
2022-11-23T03:19:55.6148000Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020000.xml (deflated 39%)
2022-11-23T03:19:55.6148744Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020009.xml (deflated 40%)
2022-11-23T03:19:55.6149514Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020015.xml (deflated 40%)
2022-11-23T03:19:55.6150269Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020022.xml (deflated 40%)
2022-11-23T03:19:55.6151003Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020029.xml (deflated 40%)
2022-11-23T03:19:55.6151750Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020035.xml (deflated 40%)
2022-11-23T03:19:55.6152510Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020042.xml (deflated 39%)
2022-11-23T03:19:55.6153260Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020051.xml (deflated 40%)
2022-11-23T03:19:55.6153992Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020058.xml (deflated 40%)
2022-11-23T03:19:55.6154741Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020105.xml (deflated 40%)
2022-11-23T03:19:55.6155577Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020113.xml (deflated 40%)
2022-11-23T03:19:55.6156331Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020120.xml (deflated 40%)
2022-11-23T03:19:55.6157133Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020127.xml (deflated 40%)
2022-11-23T03:19:55.6157965Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020136.xml (deflated 40%)
2022-11-23T03:19:55.6158694Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020143.xml (deflated 40%)
2022-11-23T03:19:55.6159446Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020149.xml (deflated 40%)
2022-11-23T03:19:55.6160201Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020158.xml (deflated 40%)
2022-11-23T03:19:55.6160951Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020205.xml (deflated 40%)
2022-11-23T03:19:55.6161676Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020212.xml (deflated 39%)
2022-11-23T03:19:55.6162433Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020221.xml (deflated 40%)
2022-11-23T03:19:55.6163174Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020227.xml (deflated 40%)
2022-11-23T03:19:55.6164016Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020234.xml (deflated 40%)
2022-11-23T03:19:55.6164828Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020241.xml (deflated 40%)
2022-11-23T03:19:55.6165585Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020252.xml (deflated 40%)
2022-11-23T03:19:55.6166326Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020258.xml (deflated 40%)
2022-11-23T03:19:55.6167079Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020305.xml (deflated 40%)
2022-11-23T03:19:55.6167805Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020314.xml (deflated 39%)
2022-11-23T03:19:55.6168552Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020321.xml (deflated 40%)
2022-11-23T03:19:55.6169314Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020328.xml (deflated 39%)
2022-11-23T03:19:55.6170065Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020337.xml (deflated 40%)
2022-11-23T03:19:55.6170803Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020344.xml (deflated 40%)
2022-11-23T03:19:55.6171550Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020351.xml (deflated 40%)
2022-11-23T03:19:55.6172298Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020357.xml (deflated 39%)
2022-11-23T03:19:55.6173050Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020406.xml (deflated 39%)
2022-11-23T03:19:55.6173780Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020413.xml (deflated 40%)
2022-11-23T03:19:55.6174534Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020420.xml (deflated 41%)
2022-11-23T03:19:55.6175347Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020422.xml (deflated 40%)
2022-11-23T03:19:55.6176092Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020429.xml (deflated 41%)
2022-11-23T03:19:55.6177007Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020431.xml (deflated 40%)
2022-11-23T03:19:55.6177778Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123020440.xml (deflated 40%)
2022-11-23T03:19:55.6178503Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123020449.xml (deflated 39%)
2022-11-23T03:19:55.6179208Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123020451.xml (deflated 39%)
2022-11-23T03:19:55.6179875Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123020453.xml (deflated 39%)
2022-11-23T03:19:55.6180567Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123020456.xml (deflated 38%)
2022-11-23T03:19:55.6181260Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123020458.xml (deflated 39%)
2022-11-23T03:19:55.6181973Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-RendezvousEnvTest-20221123020500.xml (deflated 39%)
2022-11-23T03:19:55.6182653Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-TimeoutTest-20221123020505.xml (deflated 41%)
2022-11-23T03:19:55.6183443Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020517.xml (deflated 38%)
2022-11-23T03:19:55.6184131Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020526.xml (deflated 38%)
2022-11-23T03:19:55.6184814Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020536.xml (deflated 38%)
2022-11-23T03:19:55.6185462Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020545.xml (deflated 38%)
2022-11-23T03:19:55.6186123Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020553.xml (deflated 38%)
2022-11-23T03:19:55.6186783Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020600.xml (deflated 38%)
2022-11-23T03:19:55.6187437Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020617.xml (deflated 38%)
2022-11-23T03:19:55.6188086Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020628.xml (deflated 38%)
2022-11-23T03:19:55.6188759Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020640.xml (deflated 38%)
2022-11-23T03:19:55.6189430Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020648.xml (deflated 37%)
2022-11-23T03:19:55.6190072Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020656.xml (deflated 38%)
2022-11-23T03:19:55.6190744Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020704.xml (deflated 37%)
2022-11-23T03:19:55.6191417Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020711.xml (deflated 38%)
2022-11-23T03:19:55.6192081Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020720.xml (deflated 38%)
2022-11-23T03:19:55.6192725Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020728.xml (deflated 38%)
2022-11-23T03:19:55.6193401Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020735.xml (deflated 38%)
2022-11-23T03:19:55.6194155Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020743.xml (deflated 38%)
2022-11-23T03:19:55.6194821Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020751.xml (deflated 37%)
2022-11-23T03:19:55.6195462Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123020801.xml (deflated 38%)
2022-11-23T03:19:55.6196152Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123020809.xml (deflated 38%)
2022-11-23T03:19:55.6196851Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123020818.xml (deflated 38%)
2022-11-23T03:19:55.6197553Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123020828.xml (deflated 38%)
2022-11-23T03:19:55.6198220Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123020837.xml (deflated 38%)
2022-11-23T03:19:55.6198924Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123020847.xml (deflated 38%)
2022-11-23T03:19:55.6199620Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123020856.xml (deflated 38%)
2022-11-23T03:19:55.6200311Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123020905.xml (deflated 38%)
2022-11-23T03:19:55.6201042Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123020915.xml (deflated 41%)
2022-11-23T03:19:55.6201908Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123020925.xml (deflated 41%)
2022-11-23T03:19:55.6202715Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123020935.xml (deflated 41%)
2022-11-23T03:19:55.6203531Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123020945.xml (deflated 41%)
2022-11-23T03:19:55.6204313Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123020955.xml (deflated 41%)
2022-11-23T03:19:55.6205176Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021004.xml (deflated 42%)
2022-11-23T03:19:55.6205966Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021014.xml (deflated 41%)
2022-11-23T03:19:55.6206767Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021023.xml (deflated 41%)
2022-11-23T03:19:55.6207544Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021032.xml (deflated 42%)
2022-11-23T03:19:55.6208343Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021042.xml (deflated 44%)
2022-11-23T03:19:55.6209140Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021051.xml (deflated 45%)
2022-11-23T03:19:55.6209936Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021059.xml (deflated 43%)
2022-11-23T03:19:55.6210731Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021108.xml (deflated 43%)
2022-11-23T03:19:55.6211513Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021117.xml (deflated 45%)
2022-11-23T03:19:55.6212302Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021126.xml (deflated 46%)
2022-11-23T03:19:55.6213089Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021135.xml (deflated 46%)
2022-11-23T03:19:55.6213972Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021144.xml (deflated 46%)
2022-11-23T03:19:55.6214747Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021153.xml (deflated 44%)
2022-11-23T03:19:55.6215535Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021201.xml (deflated 46%)
2022-11-23T03:19:55.6216322Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021210.xml (deflated 46%)
2022-11-23T03:19:55.6217326Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021219.xml (deflated 44%)
2022-11-23T03:19:55.6218105Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021228.xml (deflated 44%)
2022-11-23T03:19:55.6218900Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021236.xml (deflated 42%)
2022-11-23T03:19:55.6219694Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021246.xml (deflated 42%)
2022-11-23T03:19:55.6220480Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021255.xml (deflated 42%)
2022-11-23T03:19:55.6221251Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021305.xml (deflated 45%)
2022-11-23T03:19:55.6222113Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021314.xml (deflated 44%)
2022-11-23T03:19:55.6222997Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021324.xml (deflated 41%)
2022-11-23T03:19:55.6223793Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021333.xml (deflated 44%)
2022-11-23T03:19:55.6224582Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021343.xml (deflated 41%)
2022-11-23T03:19:55.6225370Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021353.xml (deflated 41%)
2022-11-23T03:19:55.6226135Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021400.xml (deflated 41%)
2022-11-23T03:19:55.6226935Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021409.xml (deflated 41%)
2022-11-23T03:19:55.6227832Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021418.xml (deflated 42%)
2022-11-23T03:19:55.6228620Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021428.xml (deflated 42%)
2022-11-23T03:19:55.6229388Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021433.xml (deflated 42%)
2022-11-23T03:19:55.6230180Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021437.xml (deflated 42%)
2022-11-23T03:19:55.6230967Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021441.xml (deflated 42%)
2022-11-23T03:19:55.6231754Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021445.xml (deflated 42%)
2022-11-23T03:19:55.6232523Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021449.xml (deflated 42%)
2022-11-23T03:19:55.6233400Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021453.xml (deflated 42%)
2022-11-23T03:19:55.6234181Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021503.xml (deflated 41%)
2022-11-23T03:19:55.6234969Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021513.xml (deflated 42%)
2022-11-23T03:19:55.6235756Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021522.xml (deflated 41%)
2022-11-23T03:19:55.6236529Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021532.xml (deflated 41%)
2022-11-23T03:19:55.6237316Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021544.xml (deflated 42%)
2022-11-23T03:19:55.6238112Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021558.xml (deflated 41%)
2022-11-23T03:19:55.6238893Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021604.xml (deflated 42%)
2022-11-23T03:19:55.6239757Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021614.xml (deflated 42%)
2022-11-23T03:19:55.6240543Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021624.xml (deflated 42%)
2022-11-23T03:19:55.6241388Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021634.xml (deflated 42%)
2022-11-23T03:19:55.6242184Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021644.xml (deflated 41%)
2022-11-23T03:19:55.6242959Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021655.xml (deflated 42%)
2022-11-23T03:19:55.6243741Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021702.xml (deflated 41%)
2022-11-23T03:19:55.6244529Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021710.xml (deflated 42%)
2022-11-23T03:19:55.6245320Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021722.xml (deflated 42%)
2022-11-23T03:19:55.6246089Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021732.xml (deflated 42%)
2022-11-23T03:19:55.6246878Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021741.xml (deflated 43%)
2022-11-23T03:19:55.6247660Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021807.xml (deflated 44%)
2022-11-23T03:19:55.6248457Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021817.xml (deflated 42%)
2022-11-23T03:19:55.6249222Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021826.xml (deflated 42%)
2022-11-23T03:19:55.6250007Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021833.xml (deflated 41%)
2022-11-23T03:19:55.6250790Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021842.xml (deflated 40%)
2022-11-23T03:19:55.6251585Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021852.xml (deflated 41%)
2022-11-23T03:19:55.6252358Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123021902.xml (deflated 41%)
2022-11-23T03:19:55.6253200Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123021913.xml (deflated 40%)
2022-11-23T03:19:55.6253957Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123021919.xml (deflated 41%)
2022-11-23T03:19:55.6254710Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123021938.xml (deflated 42%)
2022-11-23T03:19:55.6255448Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123021940.xml (deflated 41%)
2022-11-23T03:19:55.6256205Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123022002.xml (deflated 41%)
2022-11-23T03:19:55.6257182Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123022023.xml (deflated 42%)
2022-11-23T03:19:55.6257943Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123022045.xml (deflated 41%)
2022-11-23T03:19:55.6258671Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123022106.xml (deflated 42%)
2022-11-23T03:19:55.6259414Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123022109.xml (deflated 41%)
2022-11-23T03:19:55.6260288Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123022138.xml (deflated 42%)
2022-11-23T03:19:55.6261342Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123022146.xml (deflated 42%)
2022-11-23T03:19:55.6262305Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123022154.xml (deflated 44%)
2022-11-23T03:19:55.6263275Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123022202.xml (deflated 42%)
2022-11-23T03:19:55.6264153Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLNoGPUTest-20221123022210.xml (deflated 41%)
2022-11-23T03:19:55.6264926Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022213.xml (deflated 39%)
2022-11-23T03:19:55.6265679Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022221.xml (deflated 39%)
2022-11-23T03:19:55.6266420Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022230.xml (deflated 39%)
2022-11-23T03:19:55.6267169Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022240.xml (deflated 39%)
2022-11-23T03:19:55.6267923Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022249.xml (deflated 39%)
2022-11-23T03:19:55.6268668Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022259.xml (deflated 39%)
2022-11-23T03:19:55.6269399Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022308.xml (deflated 39%)
2022-11-23T03:19:55.6270143Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022317.xml (deflated 38%)
2022-11-23T03:19:55.6270879Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022325.xml (deflated 39%)
2022-11-23T03:19:55.6271633Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022335.xml (deflated 39%)
2022-11-23T03:19:55.6272360Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022349.xml (deflated 39%)
2022-11-23T03:19:55.6273189Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022357.xml (deflated 39%)
2022-11-23T03:19:55.6273935Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022406.xml (deflated 39%)
2022-11-23T03:19:55.6274680Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022414.xml (deflated 39%)
2022-11-23T03:19:55.6275408Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022424.xml (deflated 39%)
2022-11-23T03:19:55.6276154Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022433.xml (deflated 38%)
2022-11-23T03:19:55.6276898Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022441.xml (deflated 39%)
2022-11-23T03:19:55.6277643Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022451.xml (deflated 39%)
2022-11-23T03:19:55.6278356Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123022505.xml (deflated 39%)
2022-11-23T03:19:55.6279091Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-RendezvousEnvTest-20221123022513.xml (deflated 40%)
2022-11-23T03:19:55.6279793Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-TimeoutTest-20221123022517.xml (deflated 40%)
2022-11-23T03:19:55.6280679Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20221123022535.xml (deflated 44%)
2022-11-23T03:19:55.6281606Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20221123022540.xml (deflated 44%)
2022-11-23T03:19:55.6282539Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20221123022544.xml (deflated 43%)
2022-11-23T03:19:55.6283426Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123022550.xml (deflated 42%)
2022-11-23T03:19:55.6284272Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123022559.xml (deflated 42%)
2022-11-23T03:19:55.6285095Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123022609.xml (deflated 42%)
2022-11-23T03:19:55.6286049Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123022619.xml (deflated 41%)
2022-11-23T03:19:55.6286904Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123022629.xml (deflated 42%)
2022-11-23T03:19:55.6287751Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123022638.xml (deflated 41%)
2022-11-23T03:19:55.6288569Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123022648.xml (deflated 41%)
2022-11-23T03:19:55.6289403Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123022657.xml (deflated 41%)
2022-11-23T03:19:55.6290246Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022717.xml (deflated 42%)
2022-11-23T03:19:55.6291089Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022726.xml (deflated 42%)
2022-11-23T03:19:55.6291907Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022736.xml (deflated 43%)
2022-11-23T03:19:55.6292813Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022746.xml (deflated 43%)
2022-11-23T03:19:55.6293656Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022755.xml (deflated 42%)
2022-11-23T03:19:55.6294497Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022805.xml (deflated 43%)
2022-11-23T03:19:55.6295332Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022815.xml (deflated 42%)
2022-11-23T03:19:55.6296154Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022824.xml (deflated 42%)
2022-11-23T03:19:55.6297237Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022834.xml (deflated 42%)
2022-11-23T03:19:55.6298012Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123022850.xml (deflated 40%)
2022-11-23T03:19:55.6298699Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123022854.xml (deflated 39%)
2022-11-23T03:19:55.6299370Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123022858.xml (deflated 39%)
2022-11-23T03:19:55.6300053Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123022902.xml (deflated 39%)
2022-11-23T03:19:55.6300810Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20221123022906.xml (deflated 39%)
2022-11-23T03:19:55.6301514Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20221123022910.xml (deflated 40%)
2022-11-23T03:19:55.6302218Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20221123022914.xml (deflated 40%)
2022-11-23T03:19:55.6302944Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20221123022918.xml (deflated 40%)
2022-11-23T03:19:55.6303661Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixStoreTest-20221123022922.xml (deflated 40%)
2022-11-23T03:19:55.6304381Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20221123022925.xml (deflated 39%)
2022-11-23T03:19:55.6305155Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20221123022929.xml (deflated 40%)
2022-11-23T03:19:55.6305871Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20221123022933.xml (deflated 39%)
2022-11-23T03:19:55.6306576Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20221123022937.xml (deflated 39%)
2022-11-23T03:19:55.6307301Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20221123022941.xml (deflated 40%)
2022-11-23T03:19:55.6307997Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20221123022945.xml (deflated 39%)
2022-11-23T03:19:55.6308715Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123022949.xml (deflated 39%)
2022-11-23T03:19:55.6309422Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123022953.xml (deflated 39%)
2022-11-23T03:19:55.6310108Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123022957.xml (deflated 39%)
2022-11-23T03:19:55.6310816Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123023002.xml (deflated 39%)
2022-11-23T03:19:55.6311527Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20221123023016.xml (deflated 38%)
2022-11-23T03:19:55.6312311Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20221123023020.xml (deflated 39%)
2022-11-23T03:19:55.6312979Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123023024.xml (deflated 39%)
2022-11-23T03:19:55.6313663Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123023028.xml (deflated 39%)
2022-11-23T03:19:55.6314343Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123023032.xml (deflated 38%)
2022-11-23T03:19:55.6315020Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123023037.xml (deflated 38%)
2022-11-23T03:19:55.6315681Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123023041.xml (deflated 38%)
2022-11-23T03:19:55.6316364Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123023045.xml (deflated 39%)
2022-11-23T03:19:55.6317041Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123023049.xml (deflated 38%)
2022-11-23T03:19:55.6317707Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123023055.xml (deflated 38%)
2022-11-23T03:19:55.6318451Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123023107.xml (deflated 41%)
2022-11-23T03:19:55.6319275Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123023114.xml (deflated 40%)
2022-11-23T03:19:55.6320147Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123023121.xml (deflated 40%)
2022-11-23T03:19:55.6320975Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123023129.xml (deflated 40%)
2022-11-23T03:19:55.6321765Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123023138.xml (deflated 40%)
2022-11-23T03:19:55.6322572Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123023145.xml (deflated 40%)
2022-11-23T03:19:55.6323384Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123023152.xml (deflated 40%)
2022-11-23T03:19:55.6324191Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123023200.xml (deflated 40%)
2022-11-23T03:19:55.6324981Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123023209.xml (deflated 40%)
2022-11-23T03:19:55.6325781Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221123023216.xml (deflated 40%)
2022-11-23T03:19:55.6326603Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221123023223.xml (deflated 39%)
2022-11-23T03:19:55.6327407Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221123023231.xml (deflated 39%)
2022-11-23T03:19:55.6328209Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221123023239.xml (deflated 39%)
2022-11-23T03:19:55.6328991Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221123023249.xml (deflated 39%)
2022-11-23T03:19:55.6329869Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDdpComparisonTest-20221123023307.xml (deflated 41%)
2022-11-23T03:19:55.6330784Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221123023317.xml (deflated 41%)
2022-11-23T03:19:55.6331753Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221123023327.xml (deflated 41%)
2022-11-23T03:19:55.6332624Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221123023337.xml (deflated 41%)
2022-11-23T03:19:55.6333536Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221123023346.xml (deflated 41%)
2022-11-23T03:19:55.6334425Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221123023355.xml (deflated 41%)
2022-11-23T03:19:55.6335313Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221123023405.xml (deflated 41%)
2022-11-23T03:19:55.6336179Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221123023412.xml (deflated 41%)
2022-11-23T03:19:55.6337233Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRpcTest-20221123023421.xml (deflated 40%)
2022-11-23T03:19:55.6338098Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123023433.xml (deflated 40%)
2022-11-23T03:19:55.6338967Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123023445.xml (deflated 40%)
2022-11-23T03:19:55.6340001Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123023456.xml (deflated 40%)
2022-11-23T03:19:55.6340915Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123023508.xml (deflated 40%)
2022-11-23T03:19:55.6341773Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123023519.xml (deflated 39%)
2022-11-23T03:19:55.6342638Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123023531.xml (deflated 40%)
2022-11-23T03:19:55.6343502Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123023544.xml (deflated 40%)
2022-11-23T03:19:55.6344348Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123023557.xml (deflated 40%)
2022-11-23T03:19:55.6345265Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023609.xml (deflated 43%)
2022-11-23T03:19:55.6346228Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023624.xml (deflated 42%)
2022-11-23T03:19:55.6347190Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023639.xml (deflated 42%)
2022-11-23T03:19:55.6348210Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023653.xml (deflated 43%)
2022-11-23T03:19:55.6349154Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023706.xml (deflated 44%)
2022-11-23T03:19:55.6350115Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023719.xml (deflated 43%)
2022-11-23T03:19:55.6351136Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023732.xml (deflated 43%)
2022-11-23T03:19:55.6352204Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023745.xml (deflated 43%)
2022-11-23T03:19:55.6353157Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023758.xml (deflated 43%)
2022-11-23T03:19:55.6354108Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023811.xml (deflated 43%)
2022-11-23T03:19:55.6355041Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023818.xml (deflated 43%)
2022-11-23T03:19:55.6355996Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023825.xml (deflated 43%)
2022-11-23T03:19:55.6356954Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023833.xml (deflated 43%)
2022-11-23T03:19:55.6357900Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023840.xml (deflated 43%)
2022-11-23T03:19:55.6358828Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023849.xml (deflated 43%)
2022-11-23T03:19:55.6359779Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023858.xml (deflated 42%)
2022-11-23T03:19:55.6360789Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023912.xml (deflated 43%)
2022-11-23T03:19:55.6361753Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023917.xml (deflated 42%)
2022-11-23T03:19:55.6362700Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023940.xml (deflated 43%)
2022-11-23T03:19:55.6363626Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123023956.xml (deflated 42%)
2022-11-23T03:19:55.6364575Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024010.xml (deflated 43%)
2022-11-23T03:19:55.6365525Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024018.xml (deflated 42%)
2022-11-23T03:19:55.6366470Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024029.xml (deflated 42%)
2022-11-23T03:19:55.6367399Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024040.xml (deflated 42%)
2022-11-23T03:19:55.6368348Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024051.xml (deflated 43%)
2022-11-23T03:19:55.6369297Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024055.xml (deflated 42%)
2022-11-23T03:19:55.6370248Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024109.xml (deflated 42%)
2022-11-23T03:19:55.6371194Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024122.xml (deflated 42%)
2022-11-23T03:19:55.6372189Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024136.xml (deflated 43%)
2022-11-23T03:19:55.6373138Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024149.xml (deflated 43%)
2022-11-23T03:19:55.6374081Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024202.xml (deflated 43%)
2022-11-23T03:19:55.6375027Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024216.xml (deflated 42%)
2022-11-23T03:19:55.6375960Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024229.xml (deflated 42%)
2022-11-23T03:19:55.6377112Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024243.xml (deflated 42%)
2022-11-23T03:19:55.6378078Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024256.xml (deflated 42%)
2022-11-23T03:19:55.6379028Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024310.xml (deflated 43%)
2022-11-23T03:19:55.6379971Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024323.xml (deflated 42%)
2022-11-23T03:19:55.6380982Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024337.xml (deflated 42%)
2022-11-23T03:19:55.6381946Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024350.xml (deflated 42%)
2022-11-23T03:19:55.6382898Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024404.xml (deflated 42%)
2022-11-23T03:19:55.6383836Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024417.xml (deflated 42%)
2022-11-23T03:19:55.6384762Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024430.xml (deflated 42%)
2022-11-23T03:19:55.6385706Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024441.xml (deflated 43%)
2022-11-23T03:19:55.6386656Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024455.xml (deflated 42%)
2022-11-23T03:19:55.6387613Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024506.xml (deflated 42%)
2022-11-23T03:19:55.6388558Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024517.xml (deflated 43%)
2022-11-23T03:19:55.6389485Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024530.xml (deflated 43%)
2022-11-23T03:19:55.6390426Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024544.xml (deflated 43%)
2022-11-23T03:19:55.6391376Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024551.xml (deflated 43%)
2022-11-23T03:19:55.6392322Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024559.xml (deflated 43%)
2022-11-23T03:19:55.6393335Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024607.xml (deflated 42%)
2022-11-23T03:19:55.6394283Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024614.xml (deflated 42%)
2022-11-23T03:19:55.6395230Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024624.xml (deflated 42%)
2022-11-23T03:19:55.6396177Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024634.xml (deflated 42%)
2022-11-23T03:19:55.6397107Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024643.xml (deflated 42%)
2022-11-23T03:19:55.6398106Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024653.xml (deflated 42%)
2022-11-23T03:19:55.6399052Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024702.xml (deflated 42%)
2022-11-23T03:19:55.6400001Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024712.xml (deflated 42%)
2022-11-23T03:19:55.6400948Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024722.xml (deflated 43%)
2022-11-23T03:19:55.6401951Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024735.xml (deflated 42%)
2022-11-23T03:19:55.6402915Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024749.xml (deflated 42%)
2022-11-23T03:19:55.6403865Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024756.xml (deflated 43%)
2022-11-23T03:19:55.6404861Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024810.xml (deflated 42%)
2022-11-23T03:19:55.6405795Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024828.xml (deflated 42%)
2022-11-23T03:19:55.6406738Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024847.xml (deflated 42%)
2022-11-23T03:19:55.6407687Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024854.xml (deflated 43%)
2022-11-23T03:19:55.6408634Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024905.xml (deflated 42%)
2022-11-23T03:19:55.6409580Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024913.xml (deflated 42%)
2022-11-23T03:19:55.6410510Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024920.xml (deflated 42%)
2022-11-23T03:19:55.6411462Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024934.xml (deflated 41%)
2022-11-23T03:19:55.6412410Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024950.xml (deflated 42%)
2022-11-23T03:19:55.6413424Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025005.xml (deflated 42%)
2022-11-23T03:19:55.6414351Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025019.xml (deflated 42%)
2022-11-23T03:19:55.6415302Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025040.xml (deflated 42%)
2022-11-23T03:19:55.6416252Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025103.xml (deflated 43%)
2022-11-23T03:19:55.6417387Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025108.xml (deflated 42%)
2022-11-23T03:19:55.6418340Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025131.xml (deflated 42%)
2022-11-23T03:19:55.6419339Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025152.xml (deflated 42%)
2022-11-23T03:19:55.6420287Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025211.xml (deflated 42%)
2022-11-23T03:19:55.6421236Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025231.xml (deflated 42%)
2022-11-23T03:19:55.6422255Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025251.xml (deflated 42%)
2022-11-23T03:19:55.6423195Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025310.xml (deflated 42%)
2022-11-23T03:19:55.6424147Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025331.xml (deflated 42%)
2022-11-23T03:19:55.6425097Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025355.xml (deflated 42%)
2022-11-23T03:19:55.6426045Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025415.xml (deflated 42%)
2022-11-23T03:19:55.6426975Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025439.xml (deflated 42%)
2022-11-23T03:19:55.6427921Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025450.xml (deflated 43%)
2022-11-23T03:19:55.6428898Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221123025502.xml (deflated 42%)
2022-11-23T03:19:55.6429886Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221123025513.xml (deflated 43%)
2022-11-23T03:19:55.6430870Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221123025525.xml (deflated 43%)
2022-11-23T03:19:55.6431758Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper/TEST-CheckpointWrapperTest-20221123025544.xml (deflated 78%)
2022-11-23T03:19:55.6432612Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint/TEST-TestDistributedCheckpoint-20221123025554.xml (deflated 64%)
2022-11-23T03:19:55.6433474Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_apply/TEST-TestApply-20221123025604.xml (deflated 67%)
2022-11-23T03:19:55.6434237Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint/TEST-TestFSDPCheckpoint-20221123025614.xml (deflated 93%)
2022-11-23T03:19:55.6435008Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestClipGradNorm-20221123025624.xml (deflated 59%)
2022-11-23T03:19:55.6435778Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_comm/TEST-TestCommunication-20221123025633.xml (deflated 92%)
2022-11-23T03:19:55.6436566Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks/TEST-TestCommunicationHooks-20221123025643.xml (deflated 93%)
2022-11-23T03:19:55.6437350Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParityWithDDP-20221123025653.xml (deflated 95%)
2022-11-23T03:19:55.6438067Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestHooks-20221123025653.xml (deflated 84%)
2022-11-23T03:19:55.6438775Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestNoGrad-20221123025653.xml (deflated 67%)
2022-11-23T03:19:55.6439498Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParamInit-20221123025653.xml (deflated 65%)
2022-11-23T03:19:55.6440312Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order/TEST-TestFSDPExecOrder-20221123030035.xml (deflated 86%)
2022-11-23T03:19:55.6441080Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_flatten_params/TEST-TestFlattenParams-20221123030045.xml (deflated 83%)
2022-11-23T03:19:55.6441948Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights/TEST-TestFreezingWeights-20221123030055.xml (deflated 88%)
2022-11-23T03:19:55.6442744Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_fx/TEST-TestSymbolicTracing-20221123030104.xml (deflated 45%)
2022-11-23T03:19:55.6443495Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc/TEST-TestGradAcc-20221123030114.xml (deflated 94%)
2022-11-23T03:19:55.6444262Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules/TEST-TestFSDPIgnoredModules-20221123030124.xml (deflated 79%)
2022-11-23T03:19:55.6445020Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_input/TEST-TestInput-20221123030134.xml (deflated 63%)
2022-11-23T03:19:55.6445751Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_memory/TEST-TestFSDPMemory-20221123030143.xml (deflated 62%)
2022-11-23T03:19:55.6446522Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_meta/TEST-TestFSDPWithMetaDevice-20221123030153.xml (deflated 89%)
2022-11-23T03:19:55.6447260Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_misc/TEST-TestFSDPMisc-20221123030203.xml (deflated 84%)
2022-11-23T03:19:55.6448082Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionSharded-20221123030215.xml (deflated 96%)
2022-11-23T03:19:55.6448981Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionUnsharded-20221123030215.xml (deflated 72%)
2022-11-23T03:19:55.6449826Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward/TEST-TestMultiForward-20221123030223.xml (deflated 44%)
2022-11-23T03:19:55.6450641Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20221123030232.xml (deflated 47%)
2022-11-23T03:19:55.6451431Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state/TEST-TestFSDPOptimState-20221123030242.xml (deflated 95%)
2022-11-23T03:19:55.6452259Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20221123030252.xml (deflated 45%)
2022-11-23T03:19:55.6453183Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20221123030252.xml (deflated 94%)
2022-11-23T03:19:55.6453981Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_pure_fp16/TEST-TestPureFP16-20221123030631.xml (deflated 97%)
2022-11-23T03:19:55.6454742Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardGradScaler-20221123031049.xml (deflated 70%)
2022-11-23T03:19:55.6455631Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardedGradScalerParityWithDDP-20221123031049.xml (deflated 88%)
2022-11-23T03:19:55.6456488Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict/TEST-TestFSDPStateDict-20221123031059.xml (deflated 97%)
2022-11-23T03:19:55.6457489Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParams-20221123031109.xml (deflated 95%)
2022-11-23T03:19:55.6458332Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParamsNoShard-20221123031109.xml (deflated 44%)
2022-11-23T03:19:55.6459180Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_tp_integration/TEST-TestTPFSDPIntegration-20221123031119.xml (deflated 82%)
2022-11-23T03:19:55.6459961Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal/TEST-TestTraversal-20221123031128.xml (deflated 44%)
2022-11-23T03:19:55.6460728Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven/TEST-TestUnevenParamShard-20221123031138.xml (deflated 43%)
2022-11-23T03:19:55.6461603Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsFQNs-20221123031148.xml (deflated 58%)
2022-11-23T03:19:55.6462563Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsMultipleParamGroups-20221123031148.xml (deflated 86%)
2022-11-23T03:19:55.6463515Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsParamAccess-20221123031148.xml (deflated 46%)
2022-11-23T03:19:55.6464443Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsUnshardReshard-20221123031148.xml (deflated 80%)
2022-11-23T03:19:55.6465355Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsWriteback-20221123031148.xml (deflated 70%)
2022-11-23T03:19:55.6466180Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestGetSubmoduleToStates-20221123031205.xml (deflated 44%)
2022-11-23T03:19:55.6466917Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestUtils-20221123031205.xml (deflated 77%)
2022-11-23T03:19:55.6467624Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestAutoWrap-20221123031215.xml (deflated 89%)
2022-11-23T03:19:55.6468333Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestFSDPWrap-20221123031215.xml (deflated 93%)
2022-11-23T03:19:55.6469113Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_checkpoint/TEST-TestDistributedCheckpointing-20221123031225.xml (deflated 62%)
2022-11-23T03:19:55.6469974Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_checkpoint/TEST-TestDistributedFailure-20221123031225.xml (deflated 83%)
2022-11-23T03:19:55.6470859Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedReshardOnLoad-20221123031235.xml (deflated 75%)
2022-11-23T03:19:55.6471796Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoad-20221123031235.xml (deflated 44%)
2022-11-23T03:19:55.6472881Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20221123031235.xml (deflated 45%)
2022-11-23T03:19:55.6473842Z   adding: test/test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20221123031245.xml (deflated 71%)
2022-11-23T03:19:55.6474672Z   adding: test/test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20221123031245.xml (deflated 85%)
2022-11-23T03:19:55.6475482Z   adding: test/test-reports/python-unittest/distributed._shard.sharding_plan.test_sharding_plan/TEST-TestShardingPlan-20221123031254.xml (deflated 79%)
2022-11-23T03:19:55.6476354Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype/TEST-TestShardedTensorMegatronLinear-20221123031304.xml (deflated 45%)
2022-11-23T03:19:55.6477301Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalShards-20221123031314.xml (deflated 88%)
2022-11-23T03:19:55.6478217Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestCreateTensorFromParams-20221123031314.xml (deflated 44%)
2022-11-23T03:19:55.6479058Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestLocalTensor-20221123031314.xml (deflated 64%)
2022-11-23T03:19:55.6479869Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestModuleHookApi-20221123031314.xml (deflated 62%)
2022-11-23T03:19:55.6480711Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardMetadata-20221123031314.xml (deflated 62%)
2022-11-23T03:19:55.6481537Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardParameter-20221123031314.xml (deflated 64%)
2022-11-23T03:19:55.6482347Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardTensor-20221123031314.xml (deflated 64%)
2022-11-23T03:19:55.6483182Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorChunked-20221123031314.xml (deflated 91%)
2022-11-23T03:19:55.6484044Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorCustomOps-20221123031314.xml (deflated 73%)
2022-11-23T03:19:55.6484938Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorEnumerable-20221123031314.xml (deflated 89%)
2022-11-23T03:19:55.6485858Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalTensor-20221123031314.xml (deflated 65%)
2022-11-23T03:19:55.6486759Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorMetadata-20221123031314.xml (deflated 45%)
2022-11-23T03:19:55.6487580Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard/TEST-TestReshard-20221123031338.xml (deflated 64%)
2022-11-23T03:19:55.6488417Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk/TEST-TestShardedTensorChunkOps-20221123031347.xml (deflated 63%)
2022-11-23T03:19:55.6489331Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops/TEST-TestShardedTensorElementWiseOps-20221123031357.xml (deflated 77%)
2022-11-23T03:19:55.6490220Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding/TEST-TestShardedEmbedding-20221123031407.xml (deflated 63%)
2022-11-23T03:19:55.6491078Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag/TEST-TestShardedEmbeddingBag-20221123031416.xml (deflated 64%)
2022-11-23T03:19:55.6492015Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp/TEST-TestShardedTensorBinaryOps-20221123031426.xml (deflated 77%)
2022-11-23T03:19:55.6492878Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init/TEST-TestShardedTensorNNInit-20221123031436.xml (deflated 72%)
2022-11-23T03:19:55.6493735Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear/TEST-TestShardedTensorOpsLinear-20221123031446.xml (deflated 72%)
2022-11-23T03:19:55.6494618Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_matrix_ops/TEST-TestShardedTensorMatrixOps-20221123031503.xml (deflated 89%)
2022-11-23T03:19:55.6495454Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_softmax/TEST-TestShardedSoftmax-20221123031513.xml (deflated 62%)
2022-11-23T03:19:55.6496280Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim/TEST-TestShardedOptimizer-20221123031523.xml (deflated 97%)
2022-11-23T03:19:55.6497282Z   adding: test/test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorOps-20221123031913.xml (deflated 71%)
2022-11-23T03:19:55.6498097Z   adding: test/test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorReshard-20221123031913.xml (deflated 64%)
2022-11-23T03:19:55.6498841Z   adding: test/test-reports/python-unittest/test_cuda_primary_ctx/TEST-TestCudaPrimaryCtx-20221123031933.xml (deflated 41%)
2022-11-23T03:19:55.6499624Z   adding: test/test-reports/python-unittest/test_cuda_primary_ctx/TEST-TestCudaPrimaryCtx-20221123031938.xml (deflated 40%)
2022-11-23T03:19:55.6500354Z   adding: test/test-reports/python-unittest/test_cuda_primary_ctx/TEST-TestCudaPrimaryCtx-20221123031943.xml (deflated 40%)
2022-11-23T03:19:55.6537219Z ##[group]Run # Remove any previous test reports if they exist
2022-11-23T03:19:55.6537621Z # Remove any previous test reports if they exist
2022-11-23T03:19:55.6537946Z rm -f usage-log-*.zip
2022-11-23T03:19:55.6538415Z # this workflow is also run in bazel build test, but we dont generate usage reports for it
2022-11-23T03:19:55.6538807Z # so check to see if the file exists first
2022-11-23T03:19:55.6539101Z if [ -f 'usage_log.txt' ]; then
2022-11-23T03:19:55.6539434Z     zip "usage-log-${FILE_SUFFIX}.zip" 'usage_log.txt'
2022-11-23T03:19:55.6539729Z fi
2022-11-23T03:19:55.6552230Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T03:19:55.6552513Z env:
2022-11-23T03:19:55.6552751Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:19:55.6552990Z   GPU_FLAG: --gpus all
2022-11-23T03:19:55.6553338Z   DOCKER_CONTAINER_ID: bd7ceb9753ed791b4138e1687453f278a68449d77e2fd64dd747f8de67c8d5c5
2022-11-23T03:19:55.6553961Z   FILE_SUFFIX: test-multigpu-1-1-linux.16xlarge.nvidia.gpu_9655554784
2022-11-23T03:19:55.6554290Z ##[endgroup]
2022-11-23T03:19:55.7430011Z   adding: usage_log.txt (deflated 95%)
2022-11-23T03:19:55.7475330Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-11-23T03:19:55.7475627Z with:
2022-11-23T03:19:55.7475892Z   s3-prefix: pytorch/pytorch/3528394938/1/artifact

2022-11-23T03:19:55.7476188Z   retention-days: 14
2022-11-23T03:19:55.7476457Z   if-no-files-found: warn
2022-11-23T03:19:55.7476712Z   path: test-jsons-*.zip
2022-11-23T03:19:55.7476962Z   name: artifact
2022-11-23T03:19:55.7477491Z   s3-bucket: gha-artifacts
2022-11-23T03:19:55.7477608Z   region: us-east-1
2022-11-23T03:19:55.7477884Z env:
2022-11-23T03:19:55.7478064Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:19:55.7478320Z   GPU_FLAG: --gpus all
2022-11-23T03:19:55.7478681Z   DOCKER_CONTAINER_ID: bd7ceb9753ed791b4138e1687453f278a68449d77e2fd64dd747f8de67c8d5c5
2022-11-23T03:19:55.7479183Z ##[endgroup]
2022-11-23T03:19:56.1988127Z NOTE: s3-prefix specified, ignoring name parameter
2022-11-23T03:19:56.1988565Z With the provided path, there will be 1 file uploaded
2022-11-23T03:19:56.1988953Z Uploading to s3 prefix: pytorch/pytorch/3528394938/1/artifact
2022-11-23T03:19:56.1999824Z Starting upload of test-jsons-test-multigpu-1-1-linux.16xlarge.nvidia.gpu_9655554784.zip
2022-11-23T03:19:56.3243723Z Finished upload of test-jsons-test-multigpu-1-1-linux.16xlarge.nvidia.gpu_9655554784.zip
2022-11-23T03:19:56.3399512Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-11-23T03:19:56.3399803Z with:
2022-11-23T03:19:56.3400071Z   s3-prefix: pytorch/pytorch/3528394938/1/artifact

2022-11-23T03:19:56.3400373Z   retention-days: 14
2022-11-23T03:19:56.3400640Z   if-no-files-found: error
2022-11-23T03:19:56.3400917Z   path: test-reports-*.zip
2022-11-23T03:19:56.3401174Z   name: artifact
2022-11-23T03:19:56.3401425Z   s3-bucket: gha-artifacts
2022-11-23T03:19:56.3401668Z   region: us-east-1
2022-11-23T03:19:56.3401896Z env:
2022-11-23T03:19:56.3402147Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:19:56.3402388Z   GPU_FLAG: --gpus all
2022-11-23T03:19:56.3402750Z   DOCKER_CONTAINER_ID: bd7ceb9753ed791b4138e1687453f278a68449d77e2fd64dd747f8de67c8d5c5
2022-11-23T03:19:56.3403096Z ##[endgroup]
2022-11-23T03:19:56.7866600Z NOTE: s3-prefix specified, ignoring name parameter
2022-11-23T03:19:56.7867018Z With the provided path, there will be 1 file uploaded
2022-11-23T03:19:56.7867393Z Uploading to s3 prefix: pytorch/pytorch/3528394938/1/artifact
2022-11-23T03:19:56.7879614Z Starting upload of test-reports-test-multigpu-1-1-linux.16xlarge.nvidia.gpu_9655554784.zip
2022-11-23T03:19:57.0116530Z Finished upload of test-reports-test-multigpu-1-1-linux.16xlarge.nvidia.gpu_9655554784.zip
2022-11-23T03:19:57.0309606Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-11-23T03:19:57.0309913Z with:
2022-11-23T03:19:57.0310194Z   s3-prefix: pytorch/pytorch/3528394938/1/artifact

2022-11-23T03:19:57.0310482Z   retention-days: 14
2022-11-23T03:19:57.0310770Z   if-no-files-found: ignore
2022-11-23T03:19:57.0311048Z   path: usage-log-*.zip
2022-11-23T03:19:57.0311282Z   name: artifact
2022-11-23T03:19:57.0311533Z   s3-bucket: gha-artifacts
2022-11-23T03:19:57.0311793Z   region: us-east-1
2022-11-23T03:19:57.0312006Z env:
2022-11-23T03:19:57.0312245Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:19:57.0312606Z   GPU_FLAG: --gpus all
2022-11-23T03:19:57.0312860Z   DOCKER_CONTAINER_ID: bd7ceb9753ed791b4138e1687453f278a68449d77e2fd64dd747f8de67c8d5c5
2022-11-23T03:19:57.0313204Z ##[endgroup]
2022-11-23T03:19:57.4839773Z NOTE: s3-prefix specified, ignoring name parameter
2022-11-23T03:19:57.4840213Z With the provided path, there will be 1 file uploaded
2022-11-23T03:19:57.4840591Z Uploading to s3 prefix: pytorch/pytorch/3528394938/1/artifact
2022-11-23T03:19:57.4851499Z Starting upload of usage-log-test-multigpu-1-1-linux.16xlarge.nvidia.gpu_9655554784.zip
2022-11-23T03:19:57.6868424Z Finished upload of usage-log-test-multigpu-1-1-linux.16xlarge.nvidia.gpu_9655554784.zip
2022-11-23T03:19:57.7021816Z ##[group]Run # shellcheck disable=SC2156
2022-11-23T03:19:57.7022151Z # shellcheck disable=SC2156
2022-11-23T03:19:57.7022549Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \;
2022-11-23T03:19:57.7037101Z shell: /usr/bin/bash -e {0}
2022-11-23T03:19:57.7037354Z env:
2022-11-23T03:19:57.7037598Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:19:57.7037850Z   GPU_FLAG: --gpus all
2022-11-23T03:19:57.7038214Z   DOCKER_CONTAINER_ID: bd7ceb9753ed791b4138e1687453f278a68449d77e2fd64dd747f8de67c8d5c5
2022-11-23T03:19:57.7038559Z ##[endgroup]
2022-11-23T03:19:58.0263586Z ##[group]Run set -x
2022-11-23T03:19:58.0264084Z set -x
2022-11-23T03:19:58.0264382Z python3 -m pip install -r requirements.txt
2022-11-23T03:19:58.0264737Z python3 -m pip install boto3==1.19.12
2022-11-23T03:19:58.0265153Z python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-11-23T03:19:58.0280478Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T03:19:58.0280788Z env:
2022-11-23T03:19:58.0281049Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:19:58.0281307Z   GPU_FLAG: --gpus all
2022-11-23T03:19:58.0281684Z   DOCKER_CONTAINER_ID: bd7ceb9753ed791b4138e1687453f278a68449d77e2fd64dd747f8de67c8d5c5
2022-11-23T03:19:58.0282074Z   AWS_DEFAULT_REGION: us-east-1
2022-11-23T03:19:58.0282325Z   BRANCH: master
2022-11-23T03:19:58.0282593Z   TEST_CONFIG: multigpu
2022-11-23T03:19:58.0282854Z   SHARD_NUMBER: 1
2022-11-23T03:19:58.0283165Z   BUILD_ENVIRONMENT: linux-bionic-cuda11.6-py3.9-gcc7
2022-11-23T03:19:58.0283542Z   PR_NUMBER: 
2022-11-23T03:19:58.0283813Z   PYTORCH_RETRY_TEST_CASES: 1
2022-11-23T03:19:58.0284087Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2022-11-23T03:19:58.0284404Z   SHA1: 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T03:19:58.0284682Z   TAG: 
2022-11-23T03:19:58.0284900Z   WORKFLOW_ID: 3528394938
2022-11-23T03:19:58.0285342Z   GITHUB_TOKEN: ***
2022-11-23T03:19:58.0285629Z   GHA_WORKFLOW_JOB_ID: 9655554784
2022-11-23T03:19:58.0285875Z ##[endgroup]
2022-11-23T03:19:58.0320583Z + python3 -m pip install -r requirements.txt
2022-11-23T03:19:58.3337309Z Defaulting to user installation because normal site-packages is not writeable
2022-11-23T03:19:58.4232295Z Collecting astunparse
2022-11-23T03:19:58.4388737Z   Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
2022-11-23T03:19:58.4736831Z Collecting expecttest
2022-11-23T03:19:58.4782075Z   Downloading expecttest-0.1.4-py3-none-any.whl (6.5 kB)
2022-11-23T03:19:58.5197663Z Collecting future
2022-11-23T03:19:58.5245381Z   Downloading future-0.18.2.tar.gz (829 kB)
2022-11-23T03:20:00.4759402Z Collecting hypothesis
2022-11-23T03:20:00.4864821Z   Downloading hypothesis-6.58.0-py3-none-any.whl (396 kB)
2022-11-23T03:20:01.3298803Z Collecting numpy
2022-11-23T03:20:01.3384610Z   Downloading numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)
2022-11-23T03:20:01.6771806Z Requirement already satisfied: psutil in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 7)) (5.9.1)
2022-11-23T03:20:01.8087523Z Collecting pyyaml
2022-11-23T03:20:01.8147838Z   Downloading PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (596 kB)
2022-11-23T03:20:01.8501858Z Requirement already satisfied: requests in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 9)) (2.26.0)
2022-11-23T03:20:01.8697681Z Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from -r requirements.txt (line 10)) (49.1.3)
2022-11-23T03:20:01.9366147Z Collecting six
2022-11-23T03:20:01.9412398Z   Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
2022-11-23T03:20:01.9791351Z Collecting types-dataclasses
2022-11-23T03:20:01.9834956Z   Downloading types_dataclasses-0.6.6-py3-none-any.whl (2.9 kB)
2022-11-23T03:20:02.0299411Z Collecting typing_extensions
2022-11-23T03:20:02.0343381Z   Downloading typing_extensions-4.4.0-py3-none-any.whl (26 kB)
2022-11-23T03:20:02.0942714Z Collecting sympy
2022-11-23T03:20:02.1009322Z   Downloading sympy-1.10.1-py3-none-any.whl (6.4 MB)
2022-11-23T03:20:02.2950322Z Collecting filelock
2022-11-23T03:20:02.2996478Z   Downloading filelock-3.8.0-py3-none-any.whl (10 kB)
2022-11-23T03:20:02.4011842Z Collecting networkx
2022-11-23T03:20:02.4120940Z   Downloading networkx-2.6.3-py3-none-any.whl (1.9 MB)
2022-11-23T03:20:02.5438689Z Collecting jinja2
2022-11-23T03:20:02.5483993Z   Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
2022-11-23T03:20:02.6503481Z Collecting wheel<1.0,>=0.23.0
2022-11-23T03:20:02.6548751Z   Downloading wheel-0.38.4-py3-none-any.whl (36 kB)
2022-11-23T03:20:02.7088114Z Collecting attrs>=19.2.0
2022-11-23T03:20:02.7172902Z   Downloading attrs-22.1.0-py2.py3-none-any.whl (58 kB)
2022-11-23T03:20:02.8041039Z Collecting sortedcontainers<3.0.0,>=2.1.0
2022-11-23T03:20:02.8162984Z   Downloading sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB)
2022-11-23T03:20:02.8589478Z Collecting exceptiongroup>=1.0.0; python_version < "3.11"
2022-11-23T03:20:02.8634046Z   Downloading exceptiongroup-1.0.4-py3-none-any.whl (14 kB)
2022-11-23T03:20:02.8745615Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (1.26.12)
2022-11-23T03:20:02.8983028Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (3.4)
2022-11-23T03:20:02.8999048Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2.0.12)
2022-11-23T03:20:02.9027775Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2022.9.24)
2022-11-23T03:20:02.9323038Z Collecting mpmath>=0.19
2022-11-23T03:20:02.9393474Z   Downloading mpmath-1.2.1-py3-none-any.whl (532 kB)
2022-11-23T03:20:03.0989064Z Collecting MarkupSafe>=2.0
2022-11-23T03:20:03.1083050Z   Downloading MarkupSafe-2.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
2022-11-23T03:20:03.1190258Z Using legacy 'setup.py install' for future, since package 'wheel' is not installed.
2022-11-23T03:20:03.3177939Z Installing collected packages: six, wheel, astunparse, expecttest, future, attrs, sortedcontainers, exceptiongroup, hypothesis, numpy, pyyaml, types-dataclasses, typing-extensions, mpmath, sympy, filelock, networkx, MarkupSafe, jinja2
2022-11-23T03:20:03.3627640Z   WARNING: The script wheel is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-11-23T03:20:03.3628314Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-11-23T03:20:03.3987134Z     Running setup.py install for future: started
2022-11-23T03:20:04.0702279Z     Running setup.py install for future: finished with status 'done'
2022-11-23T03:20:04.4061237Z   WARNING: The script hypothesis is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-11-23T03:20:04.4061786Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-11-23T03:20:06.4469418Z   WARNING: The scripts f2py, f2py3 and f2py3.7 are installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-11-23T03:20:06.4470129Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-11-23T03:20:15.6995106Z   WARNING: The script isympy is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-11-23T03:20:15.6995912Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-11-23T03:20:17.0019041Z Successfully installed MarkupSafe-2.1.1 astunparse-1.6.3 attrs-22.1.0 exceptiongroup-1.0.4 expecttest-0.1.4 filelock-3.8.0 future-0.18.2 hypothesis-6.58.0 jinja2-3.1.2 mpmath-1.2.1 networkx-2.6.3 numpy-1.21.6 pyyaml-6.0 six-1.16.0 sortedcontainers-2.4.0 sympy-1.10.1 types-dataclasses-0.6.6 typing-extensions-4.4.0 wheel-0.38.4
2022-11-23T03:20:17.0880051Z + python3 -m pip install boto3==1.19.12
2022-11-23T03:20:17.3842138Z Defaulting to user installation because normal site-packages is not writeable
2022-11-23T03:20:18.3448369Z Collecting boto3==1.19.12
2022-11-23T03:20:18.3680545Z   Downloading boto3-1.19.12-py3-none-any.whl (131 kB)
2022-11-23T03:20:18.4325269Z Collecting s3transfer<0.6.0,>=0.5.0
2022-11-23T03:20:18.4375470Z   Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB)
2022-11-23T03:20:18.4895916Z Collecting jmespath<1.0.0,>=0.7.1
2022-11-23T03:20:18.4975237Z   Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
2022-11-23T03:20:19.7067314Z Collecting botocore<1.23.0,>=1.22.12
2022-11-23T03:20:19.7129775Z   Downloading botocore-1.22.12-py3-none-any.whl (8.1 MB)
2022-11-23T03:20:19.9542218Z Collecting python-dateutil<3.0.0,>=2.1
2022-11-23T03:20:19.9593030Z   Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
2022-11-23T03:20:19.9803455Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.12)
2022-11-23T03:20:20.0031794Z Requirement already satisfied: six>=1.5 in /home/ec2-user/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.16.0)
2022-11-23T03:20:20.2233188Z Installing collected packages: python-dateutil, jmespath, botocore, s3transfer, boto3
2022-11-23T03:20:21.1584097Z Successfully installed boto3-1.19.12 botocore-1.22.12 jmespath-0.10.0 python-dateutil-2.8.2 s3transfer-0.5.2
2022-11-23T03:20:21.2202811Z + python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-11-23T03:20:22.8928961Z [scribe] Scribe access token not provided, sending report via boto3...
2022-11-23T03:20:22.8929340Z 
2022-11-23T03:20:22.8929717Z ----- Historic stats comparison result ------
2022-11-23T03:20:22.8929941Z 
2022-11-23T03:20:22.8930191Z     job: linux-bionic-cuda11.6-py3.9-gcc7
2022-11-23T03:20:22.8934650Z     commit: 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T03:20:22.8934852Z 
2022-11-23T03:20:22.8935070Z Commit graph (base is most recent master ancestor with at least one S3 report):
2022-11-23T03:20:22.8935335Z 
2022-11-23T03:20:22.8935449Z     : (master)
2022-11-23T03:20:22.8935808Z     |
2022-11-23T03:20:22.8936067Z     * 1cfd3858ac (HEAD)              total time  3117.34s
2022-11-23T03:20:22.8936384Z     * 26322544b8          0 reports
2022-11-23T03:20:22.8943999Z     * 7f4b4d2827          0 reports
2022-11-23T03:20:22.8944466Z     * b50699f247          0 reports
2022-11-23T03:20:22.8944765Z     * 8bf8e4d71e          0 reports
2022-11-23T03:20:22.8945045Z     * ce856cee7e          0 reports
2022-11-23T03:20:22.8945416Z     * 391b593ca2          0 reports
2022-11-23T03:20:22.8945698Z     * 5bba783d21          0 reports
2022-11-23T03:20:22.8945967Z     * ea920a1115          0 reports
2022-11-23T03:20:22.8948214Z     * 74e62a1fef          0 reports
2022-11-23T03:20:22.8948505Z     * 00b7d8ef23          0 reports
2022-11-23T03:20:22.8948758Z     |
2022-11-23T03:20:22.8948965Z     :
2022-11-23T03:20:22.8949113Z 
2022-11-23T03:20:22.8949289Z Removed  (across    0 suites)     0 tests, totaling     0.00s
2022-11-23T03:20:22.8949656Z Modified (across    0 suites)     0 tests, totaling     0.00s
2022-11-23T03:20:22.8950028Z Added    (across  118 suites)  1139 tests, totaling +3117.34s
2022-11-23T03:20:22.9576621Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main
2022-11-23T03:20:22.9577271Z with:
2022-11-23T03:20:22.9577501Z env:
2022-11-23T03:20:22.9577756Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:20:22.9578017Z   GPU_FLAG: --gpus all
2022-11-23T03:20:22.9578394Z   DOCKER_CONTAINER_ID: bd7ceb9753ed791b4138e1687453f278a68449d77e2fd64dd747f8de67c8d5c5
2022-11-23T03:20:22.9578766Z ##[endgroup]
2022-11-23T03:20:22.9596740Z ##[group]Run set -eou pipefail
2022-11-23T03:20:22.9597054Z set -eou pipefail
2022-11-23T03:20:22.9597315Z 
2022-11-23T03:20:22.9597640Z echo "Holding runner for 2 hours until all ssh sessions have logged out"
2022-11-23T03:20:22.9597961Z for _ in $(seq 1440); do
2022-11-23T03:20:22.9598437Z     # Break if no ssh session exists anymore
2022-11-23T03:20:22.9598756Z     if [ "$(who)" = "" ]; then
2022-11-23T03:20:22.9599000Z       break
2022-11-23T03:20:22.9599294Z     fi
2022-11-23T03:20:22.9599548Z     echo "."
2022-11-23T03:20:22.9599775Z     sleep 5
2022-11-23T03:20:22.9600022Z done
2022-11-23T03:20:22.9614601Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T03:20:22.9614917Z env:
2022-11-23T03:20:22.9615176Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:20:22.9615453Z   GPU_FLAG: --gpus all
2022-11-23T03:20:22.9615952Z   DOCKER_CONTAINER_ID: bd7ceb9753ed791b4138e1687453f278a68449d77e2fd64dd747f8de67c8d5c5
2022-11-23T03:20:22.9616316Z ##[endgroup]
2022-11-23T03:20:22.9648169Z Holding runner for 2 hours until all ssh sessions have logged out
2022-11-23T03:20:22.9730413Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2022-11-23T03:20:22.9730864Z # ignore expansion of "docker ps -q" since it could be empty
2022-11-23T03:20:22.9731205Z # shellcheck disable=SC2046
2022-11-23T03:20:22.9731784Z docker stop $(docker ps -q) || true
2022-11-23T03:20:22.9731973Z # Prune all of the docker images
2022-11-23T03:20:22.9732265Z docker system prune -af
2022-11-23T03:20:22.9747739Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T03:20:22.9747929Z env:
2022-11-23T03:20:22.9748160Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:20:22.9748403Z   GPU_FLAG: --gpus all
2022-11-23T03:20:22.9748758Z   DOCKER_CONTAINER_ID: bd7ceb9753ed791b4138e1687453f278a68449d77e2fd64dd747f8de67c8d5c5
2022-11-23T03:20:22.9749106Z ##[endgroup]
2022-11-23T03:20:24.9390264Z bd7ceb9753ed
2022-11-23T03:20:25.4374623Z Deleted Containers:
2022-11-23T03:20:25.4375315Z bd7ceb9753ed791b4138e1687453f278a68449d77e2fd64dd747f8de67c8d5c5
2022-11-23T03:20:25.4375603Z 
2022-11-23T03:20:30.8825345Z Deleted Images:
2022-11-23T03:20:30.8826288Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T03:20:30.8827331Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7@sha256:3a5626edfb2c43fb24303351be75287af92426b6bb7c6df2defc98f980346c6a
2022-11-23T03:20:30.8827883Z deleted: sha256:e2c63e8434298b5b8922fe396fb22d541e83da3321f8559334df676354c6a90a
2022-11-23T03:20:30.8828295Z deleted: sha256:e97e2654456ae35786d9ff4a73ece4d85ce36ae9bd4e402e5f8c4c41a4b8cb5d
2022-11-23T03:20:30.8828873Z deleted: sha256:0191afefc9967131b7cd6196bee5a1d3a4eba8c24d3e11ff67013ecd0d244f4d
2022-11-23T03:20:30.8829205Z deleted: sha256:cd6998962c740e934e511d315fd0139a2737289173123cd7675b630fe71d0a6f
2022-11-23T03:20:30.8829606Z deleted: sha256:684c9dbbc4faf4388438a99012caaa6e9e9c3ac93f3842ff7b2f4c81c6c66866
2022-11-23T03:20:30.8830182Z deleted: sha256:be75865fc66b386df8a53dd220b7f4fa8464d0c86f06b6fa84e7d5b8fa2b5333
2022-11-23T03:20:30.8830498Z deleted: sha256:9e5281171ccc5aa329fd085f38d4831c13f47e27ea26a9243daf336fc701114a
2022-11-23T03:20:30.8830906Z deleted: sha256:0ba6072392ef0b01b99d45293e62f415e397460b4bf5a00257afb7aa9cfccb14
2022-11-23T03:20:30.8831606Z deleted: sha256:5f0fab79723550908a4149737ce5268ceacba20bc9c1aea35acdb6ff93ba4aa7
2022-11-23T03:20:30.8832037Z deleted: sha256:0c8088138816657b983280a5e4385f5c159c90b6be095bc4972290be20d46c16
2022-11-23T03:20:30.8832556Z deleted: sha256:a9cdd96267ff8adf28efa06db7d37977216a7580ca475239528fff85024f9bcb
2022-11-23T03:20:30.8832889Z deleted: sha256:9abd11e0b20ee19055f20e11ac5a4cc029eee3433686ce8ab9ffb6636269391a
2022-11-23T03:20:30.8833331Z deleted: sha256:cb16cc59b9c802a04fe3283c4a00840d0a3c24128b3620964a7aa927a757d672
2022-11-23T03:20:30.8833787Z deleted: sha256:ed27e40372acea88785f25bcd63f03a56960f00e444e3d5b22e52915e885242b
2022-11-23T03:20:30.8834199Z deleted: sha256:395dfa2cf9efd2fde511c14dbaf706e2efb3ab003af0cd725614b86f10643247
2022-11-23T03:20:30.8834774Z deleted: sha256:ca415181cb076083a9af8e85b901ee24154183e2d4c3960e21aab48260376214
2022-11-23T03:20:30.8835062Z deleted: sha256:b13fc2861b47406c24208813cb5398b911d9bae952f11ed9a411f42e221f8dfc
2022-11-23T03:20:30.8835501Z deleted: sha256:9cbf0b121bab50c1cad2d31b40f6c7c52003ba77877a2ef6d9bc87a2c0b073d2
2022-11-23T03:20:30.8836079Z deleted: sha256:60e157b04ecdbe2bce04795e0fade9ec9aae999065bd410785dcbaedd9778a19
2022-11-23T03:20:30.8836390Z deleted: sha256:5eb96691864f520823a417cd2f3278b4c2ac579490941d6c623865e478828c8b
2022-11-23T03:20:30.8836906Z deleted: sha256:e93d6940ac64ac73f178cc63066fb2c3ab041023d66146b32019cb7860511be5
2022-11-23T03:20:30.8837324Z deleted: sha256:e302e1f04c7e3031f83227f08d6987b02f39a75ac0e741754afad2dc1e265f8a
2022-11-23T03:20:30.8837861Z deleted: sha256:d82cdf793dbcd047c1843326443a1249721e7308a7c6fb3e23fe7331652e7047
2022-11-23T03:20:30.8838179Z deleted: sha256:3edb430c2f9009d4993daf017be01fe272bd3452db11c16e51f7755ac845d410
2022-11-23T03:20:30.8838766Z deleted: sha256:16e8f362c1784e16c1db6b1d0aa4449097e6d646f4c8682a122dea7c4da38aaf
2022-11-23T03:20:30.8839160Z deleted: sha256:7f58576cf19df9f3be9082f2c0ec2fc7010409b97ecb99bae66a10805d752f48
2022-11-23T03:20:30.8839687Z deleted: sha256:88688611a15825ecae20cd8c4032711d2351d2f954a9ebcd4c671b2bdb017df8
2022-11-23T03:20:30.8839994Z deleted: sha256:a46e0b74ccdcd4e2eb07727be3bc1a2c4236b1f88c65e64a50234e8a35932a80
2022-11-23T03:20:30.8840410Z deleted: sha256:b633962159aa14dfe94a149d00f90eecaba6dab960d4011bdf3667a5ee9586db
2022-11-23T03:20:30.8840892Z deleted: sha256:a05c7409499ce8c5d7ffc085772c3910c812ec835dc9145bbbb07b8b3c075235
2022-11-23T03:20:30.8841318Z deleted: sha256:0d63a7de5066f69cd9fd1af8fc47405e880de8f88f5cb16278a1f1ac94d0cd41
2022-11-23T03:20:30.8842036Z deleted: sha256:7d74b4ce1a60334100fccc0917345873714e640160622691b579d64c0ae4640f
2022-11-23T03:20:30.8842312Z deleted: sha256:33aae29ffc4791507bef289cfb1f178909f3fc97a40c618723eeec1f8f5bd80c
2022-11-23T03:20:30.8842752Z deleted: sha256:6ea72b84f0436ed1d288baf124dd38e43bbb89e746ccfd3a4ec420ddced8bbc2
2022-11-23T03:20:30.8843206Z deleted: sha256:04e33e1cfdd5a1b2409b80f5881e6cf7b1810fe975aad4ce7c97b0ff6c0e7b4e
2022-11-23T03:20:30.8843633Z deleted: sha256:df1ef30e86bf04681ecd0728263efe1e98b2eea0a228cef29bd0febfc8bdac2f
2022-11-23T03:20:30.8844207Z deleted: sha256:36a44974e500014175f5e49f50c8afa1ac9c5e8092a8ea99c3c97b7ce9c517d8
2022-11-23T03:20:30.8844800Z deleted: sha256:a31f0224d50d031823b07dbb97f256f6960c87ea3c52ebeceef98febab200451
2022-11-23T03:20:30.8845385Z deleted: sha256:c4beec84548d277aff0487a9a5a8c2b3d577421e3275f36106b778c6edbb9d53
2022-11-23T03:20:30.8845690Z deleted: sha256:bcc7df3b45729f5d1802045954e76e3407d9e07ba6f516de0895d775d00ad7f8
2022-11-23T03:20:30.8846132Z deleted: sha256:84de992a179a16ba619507ec45b04b4c0da3d3fa31cedc8f6beb5aaadd7a232a
2022-11-23T03:20:30.8846681Z deleted: sha256:5011206a0b2edc2a6c68ba41313e7f283ee7c925ab6a731f8818d01352f68596
2022-11-23T03:20:30.8846978Z deleted: sha256:46a56b12ac94daa35c90ac97d26adfde704693e34613d69fb97687aa53ae33f5
2022-11-23T03:20:30.8847418Z deleted: sha256:ed2b7a9e28b3474bc9b7e68f8158ecda88b3fa3d3ab1587898fa976922af0deb
2022-11-23T03:20:30.8847907Z deleted: sha256:4a6976746db7764bb48f2a06af1fb5f88e3646edc1c9bc0d18686d5a6350cac0
2022-11-23T03:20:30.8848591Z deleted: sha256:5e175425e3e9ec93e8c6c1b7560b49ef5e95af68ec55757902072a8dca020323
2022-11-23T03:20:30.8849052Z deleted: sha256:fb740502513c6cf883c844f03760de367c4c70d09a69b9476bcf737b4578563a
2022-11-23T03:20:30.8849453Z deleted: sha256:2c105119fc030d11b3d570ec9a83948a1fb17f138df2a3245f9566b89de51495
2022-11-23T03:20:30.8849867Z deleted: sha256:8caad6b6cba0d0ced7e21fe4b2027b8647d66b7f78c34367dd8571a0520ba2c0
2022-11-23T03:20:30.8850271Z deleted: sha256:1051db32aefad193995ca536ed99e29eed4fd0340ddda721ec11e9c4eb9e93af
2022-11-23T03:20:30.8850611Z deleted: sha256:c6b2a4553f41b3b4a3dc6a26be0020c98980bb4e7186d194901769dce6716c27
2022-11-23T03:20:30.8851049Z deleted: sha256:8faec3528fe75bb31f14d0caf8707a2fe4b70f60d7e631c2b3dbb36cd6d83dd9
2022-11-23T03:20:30.8851482Z deleted: sha256:7574bc80094251ac667e6bed9dd5a808ecf6f61f23c8d4c56a69c644d06f4e32
2022-11-23T03:20:30.8851883Z deleted: sha256:69f57fbceb1b420d7e4697e0f6514887b0805ee0059bea7d51e0a832962e74bf
2022-11-23T03:20:30.8852117Z 
2022-11-23T03:20:30.8965242Z Total reclaimed space: 18.96GB
2022-11-23T03:20:30.9034280Z Post job cleanup.
2022-11-23T03:20:30.9074258Z Post job cleanup.
2022-11-23T03:20:31.0478040Z [command]/usr/bin/git version
2022-11-23T03:20:31.0533811Z git version 2.37.1
2022-11-23T03:20:31.0600983Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/4638da11-24d9-4d86-bcf7-f064d7c1719e' before making global git config changes
2022-11-23T03:20:31.0601591Z Adding repository directory to the temporary git global config as a safe directory
2022-11-23T03:20:31.0607199Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-11-23T03:20:31.0653197Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-11-23T03:20:31.0693213Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-11-23T03:20:31.1047712Z Entering 'android/libs/fbjni'
2022-11-23T03:20:31.1093876Z Entering 'third_party/FP16'
2022-11-23T03:20:31.1140401Z Entering 'third_party/FXdiv'
2022-11-23T03:20:31.1187142Z Entering 'third_party/NNPACK'
2022-11-23T03:20:31.1233364Z Entering 'third_party/QNNPACK'
2022-11-23T03:20:31.1279954Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T03:20:31.1327009Z Entering 'third_party/XNNPACK'
2022-11-23T03:20:31.1388465Z Entering 'third_party/benchmark'
2022-11-23T03:20:31.1434110Z Entering 'third_party/cpuinfo'
2022-11-23T03:20:31.1480988Z Entering 'third_party/cub'
2022-11-23T03:20:31.1528650Z Entering 'third_party/cudnn_frontend'
2022-11-23T03:20:31.1581332Z Entering 'third_party/cutlass'
2022-11-23T03:20:31.1636180Z Entering 'third_party/eigen'
2022-11-23T03:20:31.1685386Z Entering 'third_party/fbgemm'
2022-11-23T03:20:31.1731921Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T03:20:31.1778450Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T03:20:31.1822053Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T03:20:31.1868110Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T03:20:31.1913901Z Entering 'third_party/flatbuffers'
2022-11-23T03:20:31.1963968Z Entering 'third_party/fmt'
2022-11-23T03:20:31.2010611Z Entering 'third_party/foxi'
2022-11-23T03:20:31.2056167Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T03:20:31.2103019Z Entering 'third_party/gloo'
2022-11-23T03:20:31.2150464Z Entering 'third_party/googletest'
2022-11-23T03:20:31.2197053Z Entering 'third_party/ideep'
2022-11-23T03:20:31.2241571Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T03:20:31.2290098Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T03:20:31.2344588Z Entering 'third_party/ios-cmake'
2022-11-23T03:20:31.2391727Z Entering 'third_party/ittapi'
2022-11-23T03:20:31.2438318Z Entering 'third_party/kineto'
2022-11-23T03:20:31.2484700Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T03:20:31.2530221Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T03:20:31.2578307Z Entering 'third_party/nccl/nccl'
2022-11-23T03:20:31.2626725Z Entering 'third_party/neon2sse'
2022-11-23T03:20:31.2672256Z Entering 'third_party/nlohmann'
2022-11-23T03:20:31.2720081Z Entering 'third_party/onnx'
2022-11-23T03:20:31.2780278Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T03:20:31.2827134Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T03:20:31.2876488Z Entering 'third_party/onnx-tensorrt'
2022-11-23T03:20:31.2921689Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T03:20:31.2972539Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T03:20:31.3020030Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T03:20:31.3067871Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T03:20:31.3120446Z Entering 'third_party/pocketfft'
2022-11-23T03:20:31.3166485Z Entering 'third_party/protobuf'
2022-11-23T03:20:31.3217935Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T03:20:31.3264112Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T03:20:31.3313015Z Entering 'third_party/psimd'
2022-11-23T03:20:31.3360167Z Entering 'third_party/pthreadpool'
2022-11-23T03:20:31.3407939Z Entering 'third_party/pybind11'
2022-11-23T03:20:31.3455396Z Entering 'third_party/python-enum'
2022-11-23T03:20:31.3501934Z Entering 'third_party/python-peachpy'
2022-11-23T03:20:31.3549043Z Entering 'third_party/python-six'
2022-11-23T03:20:31.3595262Z Entering 'third_party/sleef'
2022-11-23T03:20:31.3642085Z Entering 'third_party/tbb'
2022-11-23T03:20:31.3691629Z Entering 'third_party/tensorpipe'
2022-11-23T03:20:31.3737854Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T03:20:31.3785422Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T03:20:31.3830648Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T03:20:31.3888994Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T03:20:31.3935702Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T03:20:31.3984226Z Entering 'third_party/zstd'
2022-11-23T03:20:31.4051247Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-11-23T03:20:31.4083693Z http.https://github.com/.extraheader
2022-11-23T03:20:31.4093211Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2022-11-23T03:20:31.4132176Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-11-23T03:20:31.4510457Z Entering 'android/libs/fbjni'
2022-11-23T03:20:31.4540448Z http.https://github.com/.extraheader
2022-11-23T03:20:31.4578565Z Entering 'third_party/FP16'
2022-11-23T03:20:31.4605272Z http.https://github.com/.extraheader
2022-11-23T03:20:31.4645962Z Entering 'third_party/FXdiv'
2022-11-23T03:20:31.4675437Z http.https://github.com/.extraheader
2022-11-23T03:20:31.4714800Z Entering 'third_party/NNPACK'
2022-11-23T03:20:31.4742182Z http.https://github.com/.extraheader
2022-11-23T03:20:31.4783332Z Entering 'third_party/QNNPACK'
2022-11-23T03:20:31.4812643Z http.https://github.com/.extraheader
2022-11-23T03:20:31.4852170Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T03:20:31.4881081Z http.https://github.com/.extraheader
2022-11-23T03:20:31.4918202Z Entering 'third_party/XNNPACK'
2022-11-23T03:20:31.4947746Z http.https://github.com/.extraheader
2022-11-23T03:20:31.4996635Z Entering 'third_party/benchmark'
2022-11-23T03:20:31.5026441Z http.https://github.com/.extraheader
2022-11-23T03:20:31.5065544Z Entering 'third_party/cpuinfo'
2022-11-23T03:20:31.5094828Z http.https://github.com/.extraheader
2022-11-23T03:20:31.5134430Z Entering 'third_party/cub'
2022-11-23T03:20:31.5162481Z http.https://github.com/.extraheader
2022-11-23T03:20:31.5202443Z Entering 'third_party/cudnn_frontend'
2022-11-23T03:20:31.5230772Z http.https://github.com/.extraheader
2022-11-23T03:20:31.5275746Z Entering 'third_party/cutlass'
2022-11-23T03:20:31.5305305Z http.https://github.com/.extraheader
2022-11-23T03:20:31.5352591Z Entering 'third_party/eigen'
2022-11-23T03:20:31.5381179Z http.https://github.com/.extraheader
2022-11-23T03:20:31.5423305Z Entering 'third_party/fbgemm'
2022-11-23T03:20:31.5451185Z http.https://github.com/.extraheader
2022-11-23T03:20:31.5491799Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T03:20:31.5522307Z http.https://github.com/.extraheader
2022-11-23T03:20:31.5564843Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T03:20:31.5592432Z http.https://github.com/.extraheader
2022-11-23T03:20:31.5632808Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T03:20:31.5661445Z http.https://github.com/.extraheader
2022-11-23T03:20:31.5699473Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T03:20:31.5728533Z http.https://github.com/.extraheader
2022-11-23T03:20:31.5770321Z Entering 'third_party/flatbuffers'
2022-11-23T03:20:31.5800245Z http.https://github.com/.extraheader
2022-11-23T03:20:31.5842820Z Entering 'third_party/fmt'
2022-11-23T03:20:31.5871983Z http.https://github.com/.extraheader
2022-11-23T03:20:31.5909690Z Entering 'third_party/foxi'
2022-11-23T03:20:31.5938775Z http.https://github.com/.extraheader
2022-11-23T03:20:31.5978426Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T03:20:31.6007722Z http.https://github.com/.extraheader
2022-11-23T03:20:31.6048674Z Entering 'third_party/gloo'
2022-11-23T03:20:31.6079664Z http.https://github.com/.extraheader
2022-11-23T03:20:31.6119539Z Entering 'third_party/googletest'
2022-11-23T03:20:31.6148078Z http.https://github.com/.extraheader
2022-11-23T03:20:31.6187404Z Entering 'third_party/ideep'
2022-11-23T03:20:31.6215282Z http.https://github.com/.extraheader
2022-11-23T03:20:31.6253935Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T03:20:31.6283168Z http.https://github.com/.extraheader
2022-11-23T03:20:31.6325814Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T03:20:31.6352007Z http.https://github.com/.extraheader
2022-11-23T03:20:31.6401975Z Entering 'third_party/ios-cmake'
2022-11-23T03:20:31.6433565Z http.https://github.com/.extraheader
2022-11-23T03:20:31.6470117Z Entering 'third_party/ittapi'
2022-11-23T03:20:31.6498676Z http.https://github.com/.extraheader
2022-11-23T03:20:31.6535473Z Entering 'third_party/kineto'
2022-11-23T03:20:31.6562507Z http.https://github.com/.extraheader
2022-11-23T03:20:31.6599527Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T03:20:31.6628368Z http.https://github.com/.extraheader
2022-11-23T03:20:31.6670318Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T03:20:31.6698517Z http.https://github.com/.extraheader
2022-11-23T03:20:31.6742209Z Entering 'third_party/nccl/nccl'
2022-11-23T03:20:31.6770579Z http.https://github.com/.extraheader
2022-11-23T03:20:31.6810718Z Entering 'third_party/neon2sse'
2022-11-23T03:20:31.6839727Z http.https://github.com/.extraheader
2022-11-23T03:20:31.6879183Z Entering 'third_party/nlohmann'
2022-11-23T03:20:31.6906711Z http.https://github.com/.extraheader
2022-11-23T03:20:31.6945954Z Entering 'third_party/onnx'
2022-11-23T03:20:31.6974363Z http.https://github.com/.extraheader
2022-11-23T03:20:31.7026491Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T03:20:31.7055992Z http.https://github.com/.extraheader
2022-11-23T03:20:31.7095594Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T03:20:31.7124688Z http.https://github.com/.extraheader
2022-11-23T03:20:31.7169951Z Entering 'third_party/onnx-tensorrt'
2022-11-23T03:20:31.7197571Z http.https://github.com/.extraheader
2022-11-23T03:20:31.7238903Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T03:20:31.7265807Z http.https://github.com/.extraheader
2022-11-23T03:20:31.7311303Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T03:20:31.7340344Z http.https://github.com/.extraheader
2022-11-23T03:20:31.7379676Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T03:20:31.7409502Z http.https://github.com/.extraheader
2022-11-23T03:20:31.7444952Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T03:20:31.7476221Z http.https://github.com/.extraheader
2022-11-23T03:20:31.7521008Z Entering 'third_party/pocketfft'
2022-11-23T03:20:31.7549303Z http.https://github.com/.extraheader
2022-11-23T03:20:31.7590084Z Entering 'third_party/protobuf'
2022-11-23T03:20:31.7617706Z http.https://github.com/.extraheader
2022-11-23T03:20:31.7662426Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T03:20:31.7691104Z http.https://github.com/.extraheader
2022-11-23T03:20:31.7731285Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T03:20:31.7759483Z http.https://github.com/.extraheader
2022-11-23T03:20:31.7801124Z Entering 'third_party/psimd'
2022-11-23T03:20:31.7831010Z http.https://github.com/.extraheader
2022-11-23T03:20:31.7868785Z Entering 'third_party/pthreadpool'
2022-11-23T03:20:31.7896623Z http.https://github.com/.extraheader
2022-11-23T03:20:31.7933039Z Entering 'third_party/pybind11'
2022-11-23T03:20:31.7962248Z http.https://github.com/.extraheader
2022-11-23T03:20:31.8001270Z Entering 'third_party/python-enum'
2022-11-23T03:20:31.8028047Z http.https://github.com/.extraheader
2022-11-23T03:20:31.8066803Z Entering 'third_party/python-peachpy'
2022-11-23T03:20:31.8095402Z http.https://github.com/.extraheader
2022-11-23T03:20:31.8133381Z Entering 'third_party/python-six'
2022-11-23T03:20:31.8161110Z http.https://github.com/.extraheader
2022-11-23T03:20:31.8201993Z Entering 'third_party/sleef'
2022-11-23T03:20:31.8229947Z http.https://github.com/.extraheader
2022-11-23T03:20:31.8266020Z Entering 'third_party/tbb'
2022-11-23T03:20:31.8295533Z http.https://github.com/.extraheader
2022-11-23T03:20:31.8334570Z Entering 'third_party/tensorpipe'
2022-11-23T03:20:31.8363454Z http.https://github.com/.extraheader
2022-11-23T03:20:31.8401439Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T03:20:31.8430871Z http.https://github.com/.extraheader
2022-11-23T03:20:31.8470219Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T03:20:31.8498016Z http.https://github.com/.extraheader
2022-11-23T03:20:31.8537541Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T03:20:31.8566174Z http.https://github.com/.extraheader
2022-11-23T03:20:31.8605984Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T03:20:31.8632551Z http.https://github.com/.extraheader
2022-11-23T03:20:31.8669631Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T03:20:31.8697402Z http.https://github.com/.extraheader
2022-11-23T03:20:31.8740921Z Entering 'third_party/zstd'
2022-11-23T03:20:31.8768145Z http.https://github.com/.extraheader
2022-11-23T03:20:31.9096097Z Cleaning up orphan processes