2022-11-23T00:59:10.0020654Z Requested labels: linux.2xlarge 2022-11-23T00:59:10.0020724Z Job defined at: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/master 2022-11-23T00:59:10.0020749Z Waiting for a runner to pick up this job... 2022-11-23T00:59:10.3803711Z Job is about to start running on the runner: i-03a718e248ce6299b (organization) 2022-11-23T00:59:13.7242708Z Current runner version: '2.299.1' 2022-11-23T00:59:13.7248241Z Runner name: 'i-03a718e248ce6299b' 2022-11-23T00:59:13.7248768Z Runner group name: 'Default' 2022-11-23T00:59:13.7249393Z Machine name: 'ip-10-0-7-37' 2022-11-23T00:59:13.7251474Z ##[group]GITHUB_TOKEN Permissions 2022-11-23T00:59:13.7252197Z Actions: write 2022-11-23T00:59:13.7252488Z Checks: write 2022-11-23T00:59:13.7252820Z Contents: write 2022-11-23T00:59:13.7253188Z Deployments: write 2022-11-23T00:59:13.7253486Z Discussions: write 2022-11-23T00:59:13.7253849Z Issues: write 2022-11-23T00:59:13.7254178Z Metadata: read 2022-11-23T00:59:13.7254463Z Packages: write 2022-11-23T00:59:13.7254821Z Pages: write 2022-11-23T00:59:13.7255152Z PullRequests: write 2022-11-23T00:59:13.7255496Z RepositoryProjects: write 2022-11-23T00:59:13.7255861Z SecurityEvents: write 2022-11-23T00:59:13.7256223Z Statuses: write 2022-11-23T00:59:13.7256506Z ##[endgroup] 2022-11-23T00:59:13.7259650Z Secret source: Actions 2022-11-23T00:59:13.7260366Z Prepare workflow directory 2022-11-23T00:59:14.3197774Z Prepare all required actions 2022-11-23T00:59:14.3358256Z Getting action download info 2022-11-23T00:59:14.5192880Z Download action repository 'pytorch/test-infra@main' (SHA:c57ff4d9a93667a5571a80a0e92c3e2674aeedfd) 2022-11-23T00:59:14.7843952Z Download action repository 'pytorch/pytorch@master' (SHA:1cfd3858ac54fe3883534309081631a0a892ba3f) 2022-11-23T00:59:17.2888795Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2022-11-23T00:59:17.5346956Z Getting action download info 2022-11-23T00:59:17.6966421Z Download action repository 'malfet/checkout@silent-checkout' (SHA:c7b8fef48edfe1bca0044a44b1f7f7c4318a3076) 2022-11-23T00:59:17.8731532Z Getting action download info 2022-11-23T00:59:18.0541547Z Download action repository 'nick-fields/retry@7d4a37704547a311dbb66ebdf5b23ec19374a767' (SHA:7d4a37704547a311dbb66ebdf5b23ec19374a767) 2022-11-23T00:59:18.1814786Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml 2022-11-23T00:59:18.1816466Z ##[group] Inputs 2022-11-23T00:59:18.1816775Z build-environment: linux-focal-py3.7-gcc7 2022-11-23T00:59:18.1817588Z test-matrix: { include: [ { config: "default", shard: 1, num_shards: 2, runner: "linux.2xlarge" }, { config: "default", shard: 2, num_shards: 2, runner: "linux.2xlarge" }, { config: "distributed", shard: 1, num_shards: 2, runner: "linux.2xlarge" }, { config: "distributed", shard: 2, num_shards: 2, runner: "linux.2xlarge" }, { config: "functorch", shard: 1, num_shards: 1, runner: "linux.2xlarge" }, { config: "docs_test", shard: 1, num_shards: 1, runner: "linux.2xlarge" }, { config: "jit_legacy", shard: 1, num_shards: 1, runner: "linux.2xlarge" }, { config: "backwards_compat", shard: 1, num_shards: 1, runner: "linux.2xlarge" }, ]} 2022-11-23T00:59:18.1818454Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940 2022-11-23T00:59:18.1818825Z sync-tag: 2022-11-23T00:59:18.1819528Z timeout-minutes: 240 2022-11-23T00:59:18.1819761Z ##[endgroup] 2022-11-23T00:59:18.1820243Z Complete job name: linux-focal-py3.7-gcc7 / test (distributed, 2, 2, linux.2xlarge) 2022-11-23T00:59:18.2535638Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2022-11-23T00:59:18.2535961Z with: 2022-11-23T00:59:18.2536488Z github-secret: *** 2022-11-23T00:59:18.2536735Z activate-with-label: false 2022-11-23T00:59:18.2536951Z label: with-ssh 2022-11-23T00:59:18.2537179Z remove-existing-keys: true 2022-11-23T00:59:18.2537401Z env: 2022-11-23T00:59:18.2537594Z GIT_DEFAULT_BRANCH: master 2022-11-23T00:59:18.2537812Z ##[endgroup] 2022-11-23T00:59:18.3308361Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys 2022-11-23T00:59:18.3470855Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@master 2022-11-23T00:59:18.3471130Z with: 2022-11-23T00:59:18.3471335Z submodules: recursive 2022-11-23T00:59:18.3471521Z fetch-depth: 0 2022-11-23T00:59:18.3471765Z env: 2022-11-23T00:59:18.3471922Z GIT_DEFAULT_BRANCH: master 2022-11-23T00:59:18.3472201Z ##[endgroup] 2022-11-23T00:59:18.3673273Z ##[group]Run retry () { 2022-11-23T00:59:18.3673529Z retry () { 2022-11-23T00:59:18.3673745Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2022-11-23T00:59:18.3673958Z } 2022-11-23T00:59:18.3674145Z echo "${GITHUB_WORKSPACE}" 2022-11-23T00:59:18.3674351Z if [ -z "${NO_SUDO}" ]; then 2022-11-23T00:59:18.3674576Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2022-11-23T00:59:18.3674782Z else 2022-11-23T00:59:18.3674965Z  retry rm -rf "${GITHUB_WORKSPACE}" 2022-11-23T00:59:18.3675158Z fi 2022-11-23T00:59:18.3675377Z mkdir "${GITHUB_WORKSPACE}" 2022-11-23T00:59:18.3691200Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T00:59:18.3691431Z env: 2022-11-23T00:59:18.3691611Z GIT_DEFAULT_BRANCH: master 2022-11-23T00:59:18.3691804Z NO_SUDO: 2022-11-23T00:59:18.3691964Z ##[endgroup] 2022-11-23T00:59:18.3782984Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2022-11-23T00:59:20.4607719Z ##[group]Run malfet/checkout@silent-checkout 2022-11-23T00:59:20.4607971Z with: 2022-11-23T00:59:20.4608167Z ref: 1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T00:59:20.4608359Z fetch-depth: 0 2022-11-23T00:59:20.4608542Z submodules: recursive 2022-11-23T00:59:20.4608733Z quiet-checkout: true 2022-11-23T00:59:20.4608920Z repository: pytorch/pytorch 2022-11-23T00:59:20.4609240Z token: *** 2022-11-23T00:59:20.4609414Z ssh-strict: true 2022-11-23T00:59:20.4609595Z persist-credentials: true 2022-11-23T00:59:20.4609787Z clean: true 2022-11-23T00:59:20.4609960Z lfs: false 2022-11-23T00:59:20.4610131Z set-safe-directory: true 2022-11-23T00:59:20.4610311Z env: 2022-11-23T00:59:20.4610483Z GIT_DEFAULT_BRANCH: master 2022-11-23T00:59:20.4610652Z ##[endgroup] 2022-11-23T00:59:20.5710959Z Syncing repository: pytorch/pytorch 2022-11-23T00:59:20.5712414Z ##[group]Getting Git version info 2022-11-23T00:59:20.5712854Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2022-11-23T00:59:20.5713299Z [command]/usr/bin/git version 2022-11-23T00:59:20.5713480Z git version 2.37.1 2022-11-23T00:59:20.5719973Z ##[endgroup] 2022-11-23T00:59:20.5733222Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/3e44474b-e6fe-422c-8db6-8213ee6ef569' before making global git config changes 2022-11-23T00:59:20.5733646Z Adding repository directory to the temporary git global config as a safe directory 2022-11-23T00:59:20.5737034Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2022-11-23T00:59:20.5774525Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2022-11-23T00:59:20.5778855Z ##[group]Initializing the repository 2022-11-23T00:59:20.5781418Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2022-11-23T00:59:20.5940705Z hint: Using 'master' as the name for the initial branch. This default branch name 2022-11-23T00:59:20.5941261Z hint: is subject to change. To configure the initial branch name to use in all 2022-11-23T00:59:20.5941777Z hint: of your new repositories, which will suppress this warning, call: 2022-11-23T00:59:20.5942110Z hint: 2022-11-23T00:59:20.5942561Z hint: git config --global init.defaultBranch 2022-11-23T00:59:20.5943129Z hint: 2022-11-23T00:59:20.5943479Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2022-11-23T00:59:20.5943841Z hint: 'development'. The just-created branch can be renamed via this command: 2022-11-23T00:59:20.5944076Z hint: 2022-11-23T00:59:20.5944419Z hint: git branch -m 2022-11-23T00:59:20.5944789Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2022-11-23T00:59:20.5950491Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2022-11-23T00:59:20.5981636Z ##[endgroup] 2022-11-23T00:59:20.5981994Z ##[group]Disabling automatic garbage collection 2022-11-23T00:59:20.5984911Z [command]/usr/bin/git config --local gc.auto 0 2022-11-23T00:59:20.6011430Z ##[endgroup] 2022-11-23T00:59:20.6011920Z ##[group]Setting up auth 2022-11-23T00:59:20.6017946Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2022-11-23T00:59:20.6046724Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || : 2022-11-23T00:59:20.6285709Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2022-11-23T00:59:20.6314351Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || : 2022-11-23T00:59:20.6564221Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2022-11-23T00:59:20.6603837Z ##[endgroup] 2022-11-23T00:59:20.6604199Z ##[group]Fetching the repository 2022-11-23T00:59:20.6610009Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --quiet --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2022-11-23T01:00:11.3418379Z [command]/usr/bin/git rev-parse --verify --quiet 1cfd3858ac54fe3883534309081631a0a892ba3f^{object} 2022-11-23T01:00:11.3444242Z 1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T01:00:11.3448730Z ##[endgroup] 2022-11-23T01:00:11.3449344Z ##[group]Determining the checkout info 2022-11-23T01:00:11.3449909Z ##[endgroup] 2022-11-23T01:00:11.3450476Z ##[group]Checking out the ref 2022-11-23T01:00:11.3453720Z [command]/usr/bin/git checkout --quiet --force 1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T01:00:12.6189267Z ##[endgroup] 2022-11-23T01:00:12.6189799Z ##[group]Setting up auth for fetching submodules 2022-11-23T01:00:12.6195328Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2022-11-23T01:00:12.6241490Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2022-11-23T01:00:12.6268601Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2022-11-23T01:00:12.6296327Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2022-11-23T01:00:12.6321385Z ##[endgroup] 2022-11-23T01:00:12.6321737Z ##[group]Fetching submodules 2022-11-23T01:00:12.6325205Z [command]/usr/bin/git submodule sync --recursive 2022-11-23T01:00:12.6582911Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2022-11-23T01:00:12.6838412Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2022-11-23T01:00:12.6839481Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2022-11-23T01:00:12.6841493Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2022-11-23T01:00:12.6844695Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2022-11-23T01:00:12.6847616Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK' 2022-11-23T01:00:12.6850955Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2022-11-23T01:00:12.6854165Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2022-11-23T01:00:12.6857341Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2022-11-23T01:00:12.6860700Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2022-11-23T01:00:12.6864166Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub' 2022-11-23T01:00:12.6867993Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2022-11-23T01:00:12.6871429Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2022-11-23T01:00:12.6875364Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2022-11-23T01:00:12.6879243Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2022-11-23T01:00:12.6883527Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2022-11-23T01:00:12.6887359Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2022-11-23T01:00:12.6891508Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2022-11-23T01:00:12.6895944Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2022-11-23T01:00:12.6900306Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2022-11-23T01:00:12.6904969Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2022-11-23T01:00:12.6909565Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2022-11-23T01:00:12.6914548Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake' 2022-11-23T01:00:12.6919328Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2022-11-23T01:00:12.6926044Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2022-11-23T01:00:12.6930945Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2022-11-23T01:00:12.6936139Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse' 2022-11-23T01:00:12.6941394Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2022-11-23T01:00:12.6946748Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2022-11-23T01:00:12.6952358Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt' 2022-11-23T01:00:12.6957980Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2022-11-23T01:00:12.6963894Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2022-11-23T01:00:12.6969650Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2022-11-23T01:00:12.6975638Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2022-11-23T01:00:12.6981614Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2022-11-23T01:00:12.6987913Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum' 2022-11-23T01:00:12.6994198Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2022-11-23T01:00:12.7000522Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six' 2022-11-23T01:00:12.7007029Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2022-11-23T01:00:12.7013590Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb' 2022-11-23T01:00:12.7020340Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2022-11-23T01:00:12.7027083Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd' 2022-11-23T01:00:12.7053003Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2022-11-23T01:00:12.9781928Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2022-11-23T01:00:13.2224967Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2022-11-23T01:00:13.4426054Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2022-11-23T01:00:13.7603219Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'... 2022-11-23T01:00:14.0340229Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2022-11-23T01:00:16.0679155Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2022-11-23T01:00:22.2400965Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2022-11-23T01:00:22.6408344Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2022-11-23T01:00:23.1481697Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'... 2022-11-23T01:00:24.5771363Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2022-11-23T01:00:25.6597812Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2022-11-23T01:00:27.0271693Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2022-11-23T01:00:32.1913770Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2022-11-23T01:00:32.8943202Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2022-11-23T01:00:34.2747985Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2022-11-23T01:00:35.3810453Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2022-11-23T01:00:35.5859957Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2022-11-23T01:00:36.0164048Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2022-11-23T01:00:36.3289056Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2022-11-23T01:00:37.3823983Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2022-11-23T01:00:37.7947450Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'... 2022-11-23T01:00:37.9960877Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2022-11-23T01:00:38.2659808Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2022-11-23T01:00:40.0106816Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2022-11-23T01:00:40.4456172Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'... 2022-11-23T01:00:40.8568774Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2022-11-23T01:00:46.8450847Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2022-11-23T01:00:48.7154981Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'... 2022-11-23T01:00:49.1543796Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2022-11-23T01:00:49.3699196Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2022-11-23T01:00:54.6832127Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2022-11-23T01:00:55.0087679Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2022-11-23T01:00:55.2669753Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2022-11-23T01:00:56.0736969Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-enum'... 2022-11-23T01:00:56.4078125Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2022-11-23T01:00:56.7404428Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-six'... 2022-11-23T01:00:57.0407519Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2022-11-23T01:00:57.6021846Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'... 2022-11-23T01:00:59.5996320Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2022-11-23T01:01:00.0653256Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'... 2022-11-23T01:01:02.3888207Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2022-11-23T01:01:02.3987939Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2022-11-23T01:01:02.4063602Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2022-11-23T01:01:02.4272339Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2022-11-23T01:01:02.4481819Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c' 2022-11-23T01:01:02.4809869Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2022-11-23T01:01:03.0353966Z Submodule path 'third_party/XNNPACK': checked out 'ae108ef49aa5623b896fc93d4298c49d1750d9ba' 2022-11-23T01:01:03.0548310Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2022-11-23T01:01:03.1459492Z Submodule path 'third_party/cpuinfo': checked out '8ec7bd91ad0470e61cf38f618cc1f270dede599c' 2022-11-23T01:01:03.1776188Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4' 2022-11-23T01:01:03.4429208Z Submodule path 'third_party/cudnn_frontend': checked out '171a7a986f7fbd9ed71bd0cf3c7ad4f55843d6b3' 2022-11-23T01:01:03.8155060Z Submodule path 'third_party/cutlass': checked out 'b72cbf957df8cf84a6d0ff91c190ad51a9c1d24a' 2022-11-23T01:01:04.0372536Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2022-11-23T01:01:04.0791503Z Submodule path 'third_party/fbgemm': checked out '4d1738b3142a6cb0c032cd639e239566010b054a' 2022-11-23T01:01:04.0805880Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2022-11-23T01:01:04.0807751Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T01:01:04.0809810Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2022-11-23T01:01:04.0812150Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T01:01:04.0835315Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2022-11-23T01:01:05.1244893Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2022-11-23T01:01:05.6948504Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2022-11-23T01:01:06.7216344Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2022-11-23T01:01:07.0472293Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2022-11-23T01:01:07.1401508Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2022-11-23T01:01:07.1944945Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2022-11-23T01:01:07.2039351Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '1840658c184f3eeba787dae0f06c45756c1daaf5' 2022-11-23T01:01:07.2891097Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa' 2022-11-23T01:01:07.3224500Z Submodule path 'third_party/fmt': checked out '7bdf0628b1276379886c7f6dda2cef2b3b374f0b' 2022-11-23T01:01:07.3308046Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2022-11-23T01:01:07.3675100Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2022-11-23T01:01:07.3894025Z Submodule path 'third_party/gloo': checked out '4a5e339b764261d20fc409071dc7a8b8989aa195' 2022-11-23T01:01:07.4321482Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2022-11-23T01:01:07.4427607Z Submodule path 'third_party/ideep': checked out '5ddc65efe0428bbce2942b3ce5e3ce15239abe2f' 2022-11-23T01:01:07.4439776Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2022-11-23T01:01:07.4461213Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2022-11-23T01:01:15.5475832Z Submodule path 'third_party/ideep/mkl-dnn': checked out 'd19d0f795c60695bd32f894c6f01771b2dfbe24d' 2022-11-23T01:01:15.5491974Z Submodule 'third_party/oneDNN' (https://github.com/oneapi-src/oneDNN.git) registered for path 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T01:01:15.5515355Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN'... 2022-11-23T01:01:23.6195252Z Submodule path 'third_party/ideep/mkl-dnn/third_party/oneDNN': checked out '650085b2f3643aad05c629425983491d63b5c289' 2022-11-23T01:01:23.6291397Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724' 2022-11-23T01:01:23.6421381Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2022-11-23T01:01:23.7272303Z Submodule path 'third_party/kineto': checked out '6c1629809068efd78a8d56b4aa479c7ec49ae562' 2022-11-23T01:01:23.7286672Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T01:01:23.7288505Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T01:01:23.7311175Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2022-11-23T01:01:24.8557893Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2022-11-23T01:01:25.8827065Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '2591ab91c3898c9f6544fff04660276537d32ffd' 2022-11-23T01:01:25.9332201Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2022-11-23T01:01:25.9525273Z Submodule path 'third_party/nccl/nccl': checked out 'f89fd4777d2ef9229c039ff750ae21da01626f52' 2022-11-23T01:01:25.9658237Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a' 2022-11-23T01:01:26.0640807Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2022-11-23T01:01:26.2894001Z Submodule path 'third_party/onnx': checked out 'f7ee1ac60d06abe8e26c9b6bbe1e3db5286b614b' 2022-11-23T01:01:26.2919929Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2022-11-23T01:01:26.2921951Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2022-11-23T01:01:26.2945332Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2022-11-23T01:01:26.6832731Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2022-11-23T01:01:27.4674805Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2022-11-23T01:01:27.4964759Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'ffa346860b306c9bbfb341aed9c14c067751feb8' 2022-11-23T01:01:27.5105088Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f' 2022-11-23T01:01:27.5117301Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T01:01:27.5138330Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'... 2022-11-23T01:01:29.2825078Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8' 2022-11-23T01:01:29.2843377Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T01:01:29.2845299Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T01:01:29.2868377Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'... 2022-11-23T01:01:29.6919277Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'... 2022-11-23T01:01:30.5109458Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508' 2022-11-23T01:01:30.5714093Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c' 2022-11-23T01:01:30.5728247Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T01:01:30.5750215Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'... 2022-11-23T01:01:30.7835261Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2022-11-23T01:01:30.7918564Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a' 2022-11-23T01:01:31.0259959Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2022-11-23T01:01:31.0278499Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2022-11-23T01:01:31.0280714Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2022-11-23T01:01:31.0303765Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2022-11-23T01:01:31.4257963Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2022-11-23T01:01:32.4176473Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2022-11-23T01:01:32.4814549Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2022-11-23T01:01:32.4894514Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2022-11-23T01:01:32.4991864Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413' 2022-11-23T01:01:32.5300143Z Submodule path 'third_party/pybind11': checked out '80dc998efced8ceb2be59756668a7e90e8bef917' 2022-11-23T01:01:32.5386255Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7' 2022-11-23T01:01:32.5646871Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2022-11-23T01:01:32.5735438Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a' 2022-11-23T01:01:32.6137486Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff' 2022-11-23T01:01:32.7124779Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9' 2022-11-23T01:01:32.7366270Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2022-11-23T01:01:32.7380259Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2022-11-23T01:01:32.7382380Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2022-11-23T01:01:32.7384879Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2022-11-23T01:01:32.7387398Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T01:01:32.7409254Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2022-11-23T01:01:33.7640470Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2022-11-23T01:01:34.0470972Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2022-11-23T01:01:35.2907555Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2022-11-23T01:01:36.2597389Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2022-11-23T01:01:36.2736477Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2022-11-23T01:01:36.3332667Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2022-11-23T01:01:36.3588305Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2022-11-23T01:01:36.3601949Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T01:01:36.3624309Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2022-11-23T01:01:36.6431202Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2022-11-23T01:01:36.7662375Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8' 2022-11-23T01:01:36.7697577Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2022-11-23T01:01:36.7965200Z Entering 'android/libs/fbjni' 2022-11-23T01:01:36.7999314Z Entering 'third_party/FP16' 2022-11-23T01:01:36.8036019Z Entering 'third_party/FXdiv' 2022-11-23T01:01:36.8071881Z Entering 'third_party/NNPACK' 2022-11-23T01:01:36.8107796Z Entering 'third_party/QNNPACK' 2022-11-23T01:01:36.8145743Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T01:01:36.8182266Z Entering 'third_party/XNNPACK' 2022-11-23T01:01:36.8229515Z Entering 'third_party/benchmark' 2022-11-23T01:01:36.8266169Z Entering 'third_party/cpuinfo' 2022-11-23T01:01:36.8303103Z Entering 'third_party/cub' 2022-11-23T01:01:36.8339551Z Entering 'third_party/cudnn_frontend' 2022-11-23T01:01:36.8380741Z Entering 'third_party/cutlass' 2022-11-23T01:01:36.8422803Z Entering 'third_party/eigen' 2022-11-23T01:01:36.8461181Z Entering 'third_party/fbgemm' 2022-11-23T01:01:36.8497444Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T01:01:36.8532670Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T01:01:36.8567273Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T01:01:36.8602024Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T01:01:36.8638348Z Entering 'third_party/flatbuffers' 2022-11-23T01:01:36.8676839Z Entering 'third_party/fmt' 2022-11-23T01:01:36.8713589Z Entering 'third_party/foxi' 2022-11-23T01:01:36.8753285Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T01:01:36.8791508Z Entering 'third_party/gloo' 2022-11-23T01:01:36.8827650Z Entering 'third_party/googletest' 2022-11-23T01:01:36.8863687Z Entering 'third_party/ideep' 2022-11-23T01:01:36.8899432Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T01:01:36.8936502Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T01:01:36.8980227Z Entering 'third_party/ios-cmake' 2022-11-23T01:01:36.9017596Z Entering 'third_party/ittapi' 2022-11-23T01:01:36.9052659Z Entering 'third_party/kineto' 2022-11-23T01:01:36.9089378Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T01:01:36.9125547Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T01:01:36.9162170Z Entering 'third_party/nccl/nccl' 2022-11-23T01:01:36.9197825Z Entering 'third_party/neon2sse' 2022-11-23T01:01:36.9232639Z Entering 'third_party/nlohmann' 2022-11-23T01:01:36.9270812Z Entering 'third_party/onnx' 2022-11-23T01:01:36.9318909Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T01:01:36.9354459Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T01:01:36.9391921Z Entering 'third_party/onnx-tensorrt' 2022-11-23T01:01:36.9428704Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T01:01:36.9468900Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T01:01:36.9504614Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T01:01:36.9539428Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T01:01:36.9578662Z Entering 'third_party/pocketfft' 2022-11-23T01:01:36.9613912Z Entering 'third_party/protobuf' 2022-11-23T01:01:36.9653797Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T01:01:36.9688180Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T01:01:36.9724991Z Entering 'third_party/psimd' 2022-11-23T01:01:36.9760212Z Entering 'third_party/pthreadpool' 2022-11-23T01:01:36.9795855Z Entering 'third_party/pybind11' 2022-11-23T01:01:36.9831636Z Entering 'third_party/python-enum' 2022-11-23T01:01:36.9867061Z Entering 'third_party/python-peachpy' 2022-11-23T01:01:36.9905246Z Entering 'third_party/python-six' 2022-11-23T01:01:36.9941114Z Entering 'third_party/sleef' 2022-11-23T01:01:36.9976459Z Entering 'third_party/tbb' 2022-11-23T01:01:37.0014734Z Entering 'third_party/tensorpipe' 2022-11-23T01:01:37.0051828Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T01:01:37.0087341Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T01:01:37.0122313Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T01:01:37.0157307Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T01:01:37.0191574Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T01:01:37.0230686Z Entering 'third_party/zstd' 2022-11-23T01:01:37.0277809Z ##[endgroup] 2022-11-23T01:01:37.0278339Z ##[group]Persisting credentials for submodules 2022-11-23T01:01:37.0283101Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || : 2022-11-23T01:01:37.0547801Z Entering 'android/libs/fbjni' 2022-11-23T01:01:37.0584472Z Entering 'third_party/FP16' 2022-11-23T01:01:37.0618985Z Entering 'third_party/FXdiv' 2022-11-23T01:01:37.0652843Z Entering 'third_party/NNPACK' 2022-11-23T01:01:37.0688648Z Entering 'third_party/QNNPACK' 2022-11-23T01:01:37.0723571Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T01:01:37.0758116Z Entering 'third_party/XNNPACK' 2022-11-23T01:01:37.0803937Z Entering 'third_party/benchmark' 2022-11-23T01:01:37.0838263Z Entering 'third_party/cpuinfo' 2022-11-23T01:01:37.0873146Z Entering 'third_party/cub' 2022-11-23T01:01:37.0907209Z Entering 'third_party/cudnn_frontend' 2022-11-23T01:01:37.0948093Z Entering 'third_party/cutlass' 2022-11-23T01:01:37.0989177Z Entering 'third_party/eigen' 2022-11-23T01:01:37.1026364Z Entering 'third_party/fbgemm' 2022-11-23T01:01:37.1061408Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T01:01:37.1095915Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T01:01:37.1130317Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T01:01:37.1164395Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T01:01:37.1199817Z Entering 'third_party/flatbuffers' 2022-11-23T01:01:37.1236820Z Entering 'third_party/fmt' 2022-11-23T01:01:37.1271947Z Entering 'third_party/foxi' 2022-11-23T01:01:37.1305911Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T01:01:37.1340408Z Entering 'third_party/gloo' 2022-11-23T01:01:37.1377133Z Entering 'third_party/googletest' 2022-11-23T01:01:37.1412577Z Entering 'third_party/ideep' 2022-11-23T01:01:37.1446799Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T01:01:37.1483528Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T01:01:37.1526502Z Entering 'third_party/ios-cmake' 2022-11-23T01:01:37.1562420Z Entering 'third_party/ittapi' 2022-11-23T01:01:37.1596336Z Entering 'third_party/kineto' 2022-11-23T01:01:37.1632139Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T01:01:37.1666772Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T01:01:37.1702918Z Entering 'third_party/nccl/nccl' 2022-11-23T01:01:37.1737158Z Entering 'third_party/neon2sse' 2022-11-23T01:01:37.1773041Z Entering 'third_party/nlohmann' 2022-11-23T01:01:37.1810741Z Entering 'third_party/onnx' 2022-11-23T01:01:37.1857137Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T01:01:37.1891442Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T01:01:37.1928505Z Entering 'third_party/onnx-tensorrt' 2022-11-23T01:01:37.1965111Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T01:01:37.2004434Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T01:01:37.2038853Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T01:01:37.2073295Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T01:01:37.2113496Z Entering 'third_party/pocketfft' 2022-11-23T01:01:37.2147793Z Entering 'third_party/protobuf' 2022-11-23T01:01:37.2187096Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T01:01:37.2221698Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T01:01:37.2257052Z Entering 'third_party/psimd' 2022-11-23T01:01:37.2291712Z Entering 'third_party/pthreadpool' 2022-11-23T01:01:37.2327356Z Entering 'third_party/pybind11' 2022-11-23T01:01:37.2362324Z Entering 'third_party/python-enum' 2022-11-23T01:01:37.2397206Z Entering 'third_party/python-peachpy' 2022-11-23T01:01:37.2432373Z Entering 'third_party/python-six' 2022-11-23T01:01:37.2467446Z Entering 'third_party/sleef' 2022-11-23T01:01:37.2503203Z Entering 'third_party/tbb' 2022-11-23T01:01:37.2540268Z Entering 'third_party/tensorpipe' 2022-11-23T01:01:37.2578089Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T01:01:37.2613363Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T01:01:37.2647113Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T01:01:37.2682875Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T01:01:37.2715350Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T01:01:37.2751814Z Entering 'third_party/zstd' 2022-11-23T01:01:37.2798086Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url 2022-11-23T01:01:37.3062389Z Entering 'android/libs/fbjni' 2022-11-23T01:01:37.3095510Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2022-11-23T01:01:37.3110059Z Entering 'third_party/FP16' 2022-11-23T01:01:37.3143818Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2022-11-23T01:01:37.3158207Z Entering 'third_party/FXdiv' 2022-11-23T01:01:37.3195833Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2022-11-23T01:01:37.3209811Z Entering 'third_party/NNPACK' 2022-11-23T01:01:37.3241597Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2022-11-23T01:01:37.3256156Z Entering 'third_party/QNNPACK' 2022-11-23T01:01:37.3290763Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config remote.origin.url 2022-11-23T01:01:37.3305931Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T01:01:37.3339377Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2022-11-23T01:01:37.3354508Z Entering 'third_party/XNNPACK' 2022-11-23T01:01:37.3386733Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2022-11-23T01:01:37.3412069Z Entering 'third_party/benchmark' 2022-11-23T01:01:37.3444979Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2022-11-23T01:01:37.3459823Z Entering 'third_party/cpuinfo' 2022-11-23T01:01:37.3493039Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2022-11-23T01:01:37.3507795Z Entering 'third_party/cub' 2022-11-23T01:01:37.3539765Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config remote.origin.url 2022-11-23T01:01:37.3554814Z Entering 'third_party/cudnn_frontend' 2022-11-23T01:01:37.3587876Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2022-11-23T01:01:37.3608088Z Entering 'third_party/cutlass' 2022-11-23T01:01:37.3639868Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2022-11-23T01:01:37.3660285Z Entering 'third_party/eigen' 2022-11-23T01:01:37.3693070Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2022-11-23T01:01:37.3709670Z Entering 'third_party/fbgemm' 2022-11-23T01:01:37.3742750Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2022-11-23T01:01:37.3757091Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T01:01:37.3789408Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2022-11-23T01:01:37.3805317Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T01:01:37.3837755Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2022-11-23T01:01:37.3852355Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T01:01:37.3884056Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2022-11-23T01:01:37.3898266Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T01:01:37.3930777Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2022-11-23T01:01:37.3946608Z Entering 'third_party/flatbuffers' 2022-11-23T01:01:37.3979495Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2022-11-23T01:01:37.3997165Z Entering 'third_party/fmt' 2022-11-23T01:01:37.4028505Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2022-11-23T01:01:37.4045077Z Entering 'third_party/foxi' 2022-11-23T01:01:37.4076355Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2022-11-23T01:01:37.4090374Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T01:01:37.4122808Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2022-11-23T01:01:37.4137837Z Entering 'third_party/gloo' 2022-11-23T01:01:37.4170130Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2022-11-23T01:01:37.4183357Z Entering 'third_party/googletest' 2022-11-23T01:01:37.4217829Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2022-11-23T01:01:37.4233007Z Entering 'third_party/ideep' 2022-11-23T01:01:37.4266152Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2022-11-23T01:01:37.4281036Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T01:01:37.4313097Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2022-11-23T01:01:37.4329907Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T01:01:37.4362733Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/modules/third_party/oneDNN/config remote.origin.url 2022-11-23T01:01:37.4383435Z Entering 'third_party/ios-cmake' 2022-11-23T01:01:37.4415704Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config remote.origin.url 2022-11-23T01:01:37.4430712Z Entering 'third_party/ittapi' 2022-11-23T01:01:37.4462519Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2022-11-23T01:01:37.4476753Z Entering 'third_party/kineto' 2022-11-23T01:01:37.4509163Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2022-11-23T01:01:37.4523745Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T01:01:37.4556014Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2022-11-23T01:01:37.4570610Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T01:01:37.4602639Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2022-11-23T01:01:37.4617301Z Entering 'third_party/nccl/nccl' 2022-11-23T01:01:37.4649617Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2022-11-23T01:01:37.4664005Z Entering 'third_party/neon2sse' 2022-11-23T01:01:37.4696950Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config remote.origin.url 2022-11-23T01:01:37.4711204Z Entering 'third_party/nlohmann' 2022-11-23T01:01:37.4743264Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2022-11-23T01:01:37.4759377Z Entering 'third_party/onnx' 2022-11-23T01:01:37.4792450Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2022-11-23T01:01:37.4818511Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T01:01:37.4850773Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2022-11-23T01:01:37.4865522Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T01:01:37.4898209Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2022-11-23T01:01:37.4914465Z Entering 'third_party/onnx-tensorrt' 2022-11-23T01:01:37.4946234Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config remote.origin.url 2022-11-23T01:01:37.4961583Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T01:01:37.4992884Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config remote.origin.url 2022-11-23T01:01:37.5012375Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T01:01:37.5044201Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2022-11-23T01:01:37.5059178Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T01:01:37.5091791Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2022-11-23T01:01:37.5106050Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T01:01:37.5139082Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2022-11-23T01:01:37.5158915Z Entering 'third_party/pocketfft' 2022-11-23T01:01:37.5191173Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2022-11-23T01:01:37.5205732Z Entering 'third_party/protobuf' 2022-11-23T01:01:37.5238796Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2022-11-23T01:01:37.5257827Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T01:01:37.5290161Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2022-11-23T01:01:37.5305047Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T01:01:37.5338027Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2022-11-23T01:01:37.5354204Z Entering 'third_party/psimd' 2022-11-23T01:01:37.5386601Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2022-11-23T01:01:37.5401761Z Entering 'third_party/pthreadpool' 2022-11-23T01:01:37.5433551Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2022-11-23T01:01:37.5447623Z Entering 'third_party/pybind11' 2022-11-23T01:01:37.5480489Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2022-11-23T01:01:37.5495856Z Entering 'third_party/python-enum' 2022-11-23T01:01:37.5528372Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config remote.origin.url 2022-11-23T01:01:37.5542958Z Entering 'third_party/python-peachpy' 2022-11-23T01:01:37.5576110Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2022-11-23T01:01:37.5591051Z Entering 'third_party/python-six' 2022-11-23T01:01:37.5622971Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config remote.origin.url 2022-11-23T01:01:37.5637903Z Entering 'third_party/sleef' 2022-11-23T01:01:37.5670095Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2022-11-23T01:01:37.5685229Z Entering 'third_party/tbb' 2022-11-23T01:01:37.5718072Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config remote.origin.url 2022-11-23T01:01:37.5735186Z Entering 'third_party/tensorpipe' 2022-11-23T01:01:37.5768229Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2022-11-23T01:01:37.5782839Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T01:01:37.5816024Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2022-11-23T01:01:37.5830406Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T01:01:37.5862625Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2022-11-23T01:01:37.5877142Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T01:01:37.5910187Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2022-11-23T01:01:37.5925109Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T01:01:37.5958295Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2022-11-23T01:01:37.5972527Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T01:01:37.6004925Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2022-11-23T01:01:37.6022620Z Entering 'third_party/zstd' 2022-11-23T01:01:37.6057438Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config remote.origin.url 2022-11-23T01:01:37.7700552Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2022-11-23T01:01:37.7967410Z Entering 'android/libs/fbjni' 2022-11-23T01:01:37.8003488Z Entering 'third_party/FP16' 2022-11-23T01:01:37.8038566Z Entering 'third_party/FXdiv' 2022-11-23T01:01:37.8075844Z Entering 'third_party/NNPACK' 2022-11-23T01:01:37.8111954Z Entering 'third_party/QNNPACK' 2022-11-23T01:01:37.8149871Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T01:01:37.8185909Z Entering 'third_party/XNNPACK' 2022-11-23T01:01:37.8349070Z Entering 'third_party/benchmark' 2022-11-23T01:01:37.8385127Z Entering 'third_party/cpuinfo' 2022-11-23T01:01:37.8422581Z Entering 'third_party/cub' 2022-11-23T01:01:37.8459159Z Entering 'third_party/cudnn_frontend' 2022-11-23T01:01:37.8499890Z Entering 'third_party/cutlass' 2022-11-23T01:01:37.8541122Z Entering 'third_party/eigen' 2022-11-23T01:01:37.8579220Z Entering 'third_party/fbgemm' 2022-11-23T01:01:37.8613501Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T01:01:37.8649360Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T01:01:37.8685068Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T01:01:37.8719324Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T01:01:37.8754741Z Entering 'third_party/flatbuffers' 2022-11-23T01:01:37.8792405Z Entering 'third_party/fmt' 2022-11-23T01:01:37.8828276Z Entering 'third_party/foxi' 2022-11-23T01:01:37.8864401Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T01:01:37.8901294Z Entering 'third_party/gloo' 2022-11-23T01:01:37.8937609Z Entering 'third_party/googletest' 2022-11-23T01:01:37.8973267Z Entering 'third_party/ideep' 2022-11-23T01:01:37.9008212Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T01:01:37.9044005Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T01:01:37.9085033Z Entering 'third_party/ios-cmake' 2022-11-23T01:01:37.9118937Z Entering 'third_party/ittapi' 2022-11-23T01:01:37.9154923Z Entering 'third_party/kineto' 2022-11-23T01:01:37.9190728Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T01:01:37.9225311Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T01:01:37.9263175Z Entering 'third_party/nccl/nccl' 2022-11-23T01:01:37.9299438Z Entering 'third_party/neon2sse' 2022-11-23T01:01:37.9334419Z Entering 'third_party/nlohmann' 2022-11-23T01:01:37.9371799Z Entering 'third_party/onnx' 2022-11-23T01:01:37.9419102Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T01:01:37.9453636Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T01:01:37.9491187Z Entering 'third_party/onnx-tensorrt' 2022-11-23T01:01:37.9527217Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T01:01:37.9566250Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T01:01:37.9601386Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T01:01:37.9636293Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T01:01:37.9675604Z Entering 'third_party/pocketfft' 2022-11-23T01:01:37.9712849Z Entering 'third_party/protobuf' 2022-11-23T01:01:37.9751104Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T01:01:37.9785278Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T01:01:37.9822698Z Entering 'third_party/psimd' 2022-11-23T01:01:37.9858289Z Entering 'third_party/pthreadpool' 2022-11-23T01:01:37.9894179Z Entering 'third_party/pybind11' 2022-11-23T01:01:37.9929363Z Entering 'third_party/python-enum' 2022-11-23T01:01:37.9965321Z Entering 'third_party/python-peachpy' 2022-11-23T01:01:38.0000077Z Entering 'third_party/python-six' 2022-11-23T01:01:38.0035634Z Entering 'third_party/sleef' 2022-11-23T01:01:38.0070853Z Entering 'third_party/tbb' 2022-11-23T01:01:38.0107986Z Entering 'third_party/tensorpipe' 2022-11-23T01:01:38.0143914Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T01:01:38.0178532Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T01:01:38.0214347Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T01:01:38.0249745Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T01:01:38.0284616Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T01:01:38.0322856Z Entering 'third_party/zstd' 2022-11-23T01:01:38.0367609Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2022-11-23T01:01:38.0627990Z Entering 'android/libs/fbjni' 2022-11-23T01:01:38.0662571Z Entering 'third_party/FP16' 2022-11-23T01:01:38.0698438Z Entering 'third_party/FXdiv' 2022-11-23T01:01:38.0735637Z Entering 'third_party/NNPACK' 2022-11-23T01:01:38.0770381Z Entering 'third_party/QNNPACK' 2022-11-23T01:01:38.0805875Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T01:01:38.0841102Z Entering 'third_party/XNNPACK' 2022-11-23T01:01:38.0886595Z Entering 'third_party/benchmark' 2022-11-23T01:01:38.0921944Z Entering 'third_party/cpuinfo' 2022-11-23T01:01:38.0957524Z Entering 'third_party/cub' 2022-11-23T01:01:38.0994283Z Entering 'third_party/cudnn_frontend' 2022-11-23T01:01:38.1035960Z Entering 'third_party/cutlass' 2022-11-23T01:01:38.1077619Z Entering 'third_party/eigen' 2022-11-23T01:01:38.1115740Z Entering 'third_party/fbgemm' 2022-11-23T01:01:38.1151154Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T01:01:38.1185827Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T01:01:38.1251466Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T01:01:38.1258550Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T01:01:38.1295002Z Entering 'third_party/flatbuffers' 2022-11-23T01:01:38.1332270Z Entering 'third_party/fmt' 2022-11-23T01:01:38.1367977Z Entering 'third_party/foxi' 2022-11-23T01:01:38.1403123Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T01:01:38.1438433Z Entering 'third_party/gloo' 2022-11-23T01:01:38.1473209Z Entering 'third_party/googletest' 2022-11-23T01:01:38.1508996Z Entering 'third_party/ideep' 2022-11-23T01:01:38.1545670Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T01:01:38.1582222Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T01:01:38.1623965Z Entering 'third_party/ios-cmake' 2022-11-23T01:01:38.1659281Z Entering 'third_party/ittapi' 2022-11-23T01:01:38.1696468Z Entering 'third_party/kineto' 2022-11-23T01:01:38.1732461Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T01:01:38.1767700Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T01:01:38.1804140Z Entering 'third_party/nccl/nccl' 2022-11-23T01:01:38.1839716Z Entering 'third_party/neon2sse' 2022-11-23T01:01:38.1877084Z Entering 'third_party/nlohmann' 2022-11-23T01:01:38.1912862Z Entering 'third_party/onnx' 2022-11-23T01:01:38.1959212Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T01:01:38.1995785Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T01:01:38.2033709Z Entering 'third_party/onnx-tensorrt' 2022-11-23T01:01:38.2069219Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T01:01:38.2107492Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T01:01:38.2144975Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T01:01:38.2179543Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T01:01:38.2218367Z Entering 'third_party/pocketfft' 2022-11-23T01:01:38.2253399Z Entering 'third_party/protobuf' 2022-11-23T01:01:38.2290973Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T01:01:38.2326366Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T01:01:38.2362394Z Entering 'third_party/psimd' 2022-11-23T01:01:38.2396338Z Entering 'third_party/pthreadpool' 2022-11-23T01:01:38.2430935Z Entering 'third_party/pybind11' 2022-11-23T01:01:38.2465820Z Entering 'third_party/python-enum' 2022-11-23T01:01:38.2501648Z Entering 'third_party/python-peachpy' 2022-11-23T01:01:38.2537269Z Entering 'third_party/python-six' 2022-11-23T01:01:38.2572294Z Entering 'third_party/sleef' 2022-11-23T01:01:38.2607768Z Entering 'third_party/tbb' 2022-11-23T01:01:38.2645200Z Entering 'third_party/tensorpipe' 2022-11-23T01:01:38.2680948Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T01:01:38.2715472Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T01:01:38.2749397Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T01:01:38.2784446Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T01:01:38.2818265Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T01:01:38.2855672Z Entering 'third_party/zstd' 2022-11-23T01:01:38.2900019Z ##[endgroup] 2022-11-23T01:01:38.2945092Z [command]/usr/bin/git log -1 --format='%H' 2022-11-23T01:01:38.2970972Z '1cfd3858ac54fe3883534309081631a0a892ba3f' 2022-11-23T01:01:38.3093718Z Prepare all required actions 2022-11-23T01:01:38.3118114Z ##[group]Run ./.github/actions/setup-linux 2022-11-23T01:01:38.3118345Z env: 2022-11-23T01:01:38.3118552Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:01:38.3118755Z ##[endgroup] 2022-11-23T01:01:38.3133844Z ##[group]Run set -euo pipefail 2022-11-23T01:01:38.3134108Z set -euo pipefail 2022-11-23T01:01:38.3134353Z function get_ec2_metadata() { 2022-11-23T01:01:38.3134634Z  # Pulled from instance metadata endpoint for EC2 2022-11-23T01:01:38.3135011Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2022-11-23T01:01:38.3135339Z  category=$1 2022-11-23T01:01:38.3135614Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2022-11-23T01:01:38.3135858Z } 2022-11-23T01:01:38.3136092Z echo "ami-id: $(get_ec2_metadata ami-id)" 2022-11-23T01:01:38.3136412Z echo "instance-id: $(get_ec2_metadata instance-id)" 2022-11-23T01:01:38.3136706Z echo "instance-type: $(get_ec2_metadata instance-type)" 2022-11-23T01:01:38.3136984Z echo "system info $(uname -a)" 2022-11-23T01:01:38.3148539Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:01:38.3148781Z env: 2022-11-23T01:01:38.3148986Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:01:38.3149205Z ##[endgroup] 2022-11-23T01:01:38.3239696Z ami-id: ami-096198a0bccc6bad4 2022-11-23T01:01:38.3290743Z instance-id: i-03a718e248ce6299b 2022-11-23T01:01:38.3347191Z instance-type: c5.2xlarge 2022-11-23T01:01:38.3353753Z system info Linux ip-10-0-7-37.ec2.internal 4.14.252-195.483.amzn2.x86_64 #1 SMP Mon Nov 1 20:58:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux 2022-11-23T01:01:38.3368971Z ##[group]Run if systemctl is-active --quiet docker; then 2022-11-23T01:01:38.3369266Z if systemctl is-active --quiet docker; then 2022-11-23T01:01:38.3369517Z  echo "Docker daemon is running..."; 2022-11-23T01:01:38.3369711Z else 2022-11-23T01:01:38.3369946Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2022-11-23T01:01:38.3370178Z fi 2022-11-23T01:01:38.3381160Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:01:38.3381387Z env: 2022-11-23T01:01:38.3381563Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:01:38.3381736Z ##[endgroup] 2022-11-23T01:01:38.3484514Z Docker daemon is running... 2022-11-23T01:01:38.3500304Z ##[group]Run AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") 2022-11-23T01:01:38.3500692Z AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") 2022-11-23T01:01:38.3501005Z retry () { "$@" || (sleep 1 && "$@") || (sleep 2 && "$@") } 2022-11-23T01:01:38.3501424Z retry aws ecr get-login*** "$AWS_DEFAULT_REGION" | docker login --username AWS \ 2022-11-23T01:01:38.3501790Z  --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2022-11-23T01:01:38.3512733Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:01:38.3512985Z env: 2022-11-23T01:01:38.3513174Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:01:38.3513402Z AWS_RETRY_MODE: standard 2022-11-23T01:01:38.3513620Z AWS_MAX_ATTEMPTS: 5 2022-11-23T01:01:38.3513840Z AWS_DEFAULT_REGION: us-east-1 2022-11-23T01:01:38.3514054Z ##[endgroup] 2022-11-23T01:01:39.8182552Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2022-11-23T01:01:39.8182919Z Configure a credential helper to remove this warning. See 2022-11-23T01:01:39.8183323Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2022-11-23T01:01:39.8183586Z 2022-11-23T01:01:39.8183897Z Login Succeeded 2022-11-23T01:01:39.8217074Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2022-11-23T01:01:39.8217419Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2022-11-23T01:01:39.8217800Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2022-11-23T01:01:39.8229171Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:01:39.8229426Z env: 2022-11-23T01:01:39.8229632Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:01:39.8229848Z ##[endgroup] 2022-11-23T01:01:39.8316264Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2022-11-23T01:01:39.8316514Z with: 2022-11-23T01:01:39.8316832Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940 2022-11-23T01:01:39.8317150Z env: 2022-11-23T01:01:39.8317323Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:01:39.8317496Z ##[endgroup] 2022-11-23T01:01:39.8329085Z ##[group]Run retry () { "$@" || (sleep 1 && "$@") || (sleep 2 && "$@") } 2022-11-23T01:01:39.8329356Z retry () { "$@" || (sleep 1 && "$@") || (sleep 2 && "$@") } 2022-11-23T01:01:39.8329620Z # ignore output since only exit code is used for conditional 2022-11-23T01:01:39.8329898Z # only pull docker image if it's not available locally 2022-11-23T01:01:39.8330180Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2022-11-23T01:01:39.8330481Z  retry docker pull "${DOCKER_IMAGE}" 2022-11-23T01:01:39.8330680Z fi 2022-11-23T01:01:39.8341095Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:01:39.8341314Z env: 2022-11-23T01:01:39.8341488Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:01:39.8341829Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940 2022-11-23T01:01:39.8342158Z ##[endgroup] 2022-11-23T01:01:40.0641900Z 072aae4a77ed7d3a69ad5683420509c41301b940: Pulling from pytorch/pytorch-linux-focal-py3.7-gcc7 2022-11-23T01:01:40.0650603Z eaead16dc43b: Pulling fs layer 2022-11-23T01:01:40.0650872Z 96afc5db69dc: Pulling fs layer 2022-11-23T01:01:40.0651587Z 9cd06b79192a: Pulling fs layer 2022-11-23T01:01:40.0651956Z ed66d74b7c20: Pulling fs layer 2022-11-23T01:01:40.0652302Z 42439e1d77c4: Pulling fs layer 2022-11-23T01:01:40.0652678Z 80bf4ab5d594: Pulling fs layer 2022-11-23T01:01:40.0653011Z 879beb9f8e9b: Pulling fs layer 2022-11-23T01:01:40.0653360Z 3f623e97839a: Pulling fs layer 2022-11-23T01:01:40.0653672Z d845eb9c30a0: Pulling fs layer 2022-11-23T01:01:40.0653987Z 8654f94b3130: Pulling fs layer 2022-11-23T01:01:40.0654331Z 0c488367249b: Pulling fs layer 2022-11-23T01:01:40.0654682Z 0c91414faf6f: Pulling fs layer 2022-11-23T01:01:40.0654985Z 16b14aea0152: Pulling fs layer 2022-11-23T01:01:40.0655311Z a2871863f729: Pulling fs layer 2022-11-23T01:01:40.0655623Z 42439e1d77c4: Waiting 2022-11-23T01:01:40.0655943Z 879beb9f8e9b: Waiting 2022-11-23T01:01:40.0656239Z 80bf4ab5d594: Waiting 2022-11-23T01:01:40.0656567Z d703d3396c78: Pulling fs layer 2022-11-23T01:01:40.0656882Z 0c488367249b: Waiting 2022-11-23T01:01:40.0657202Z acf65ded4ba0: Pulling fs layer 2022-11-23T01:01:40.0657532Z 0c91414faf6f: Waiting 2022-11-23T01:01:40.0657764Z 3f623e97839a: Waiting 2022-11-23T01:01:40.0657937Z 19d7da72e211: Pulling fs layer 2022-11-23T01:01:40.0658121Z 8654f94b3130: Waiting 2022-11-23T01:01:40.0658290Z d845eb9c30a0: Waiting 2022-11-23T01:01:40.0658465Z cb4b60b16a23: Pulling fs layer 2022-11-23T01:01:40.0658660Z 6325a464626e: Pulling fs layer 2022-11-23T01:01:40.0658839Z a2871863f729: Waiting 2022-11-23T01:01:40.0659011Z b17aded2bf76: Pulling fs layer 2022-11-23T01:01:40.0659197Z ed66d74b7c20: Waiting 2022-11-23T01:01:40.0659369Z 16b14aea0152: Waiting 2022-11-23T01:01:40.0659798Z d703d3396c78: Waiting 2022-11-23T01:01:40.0659990Z 79f0454f85ca: Pulling fs layer 2022-11-23T01:01:40.0660177Z acf65ded4ba0: Waiting 2022-11-23T01:01:40.0660356Z 5df784aed3a4: Pulling fs layer 2022-11-23T01:01:40.0660541Z cb4b60b16a23: Waiting 2022-11-23T01:01:40.0660727Z 196c72536fc8: Pulling fs layer 2022-11-23T01:01:40.0660899Z 19d7da72e211: Waiting 2022-11-23T01:01:40.0661079Z e7cf72049767: Pulling fs layer 2022-11-23T01:01:40.0661272Z 7c4667efa9a5: Pulling fs layer 2022-11-23T01:01:40.0661441Z 6325a464626e: Waiting 2022-11-23T01:01:40.0661623Z c62a223ffc22: Pulling fs layer 2022-11-23T01:01:40.0661809Z 5df784aed3a4: Waiting 2022-11-23T01:01:40.0661982Z 824caeed1492: Pulling fs layer 2022-11-23T01:01:40.0662366Z b17aded2bf76: Waiting 2022-11-23T01:01:40.0662546Z 79f0454f85ca: Waiting 2022-11-23T01:01:40.0662715Z 1762e4b0a4dc: Pulling fs layer 2022-11-23T01:01:40.0662910Z 093e8e8bcad0: Pulling fs layer 2022-11-23T01:01:40.0663098Z e7cf72049767: Waiting 2022-11-23T01:01:40.0663255Z 196c72536fc8: Waiting 2022-11-23T01:01:40.0663436Z 133752b42076: Pulling fs layer 2022-11-23T01:01:40.0663627Z 516276b2131d: Pulling fs layer 2022-11-23T01:01:40.0663836Z 7c4667efa9a5: Waiting 2022-11-23T01:01:40.0664006Z 3507de6a25b1: Pulling fs layer 2022-11-23T01:01:40.0664200Z 49880d42ecce: Pulling fs layer 2022-11-23T01:01:40.0664386Z c62a223ffc22: Waiting 2022-11-23T01:01:40.0664556Z 4694407ca3de: Pulling fs layer 2022-11-23T01:01:40.0664749Z b2c751e637f5: Pulling fs layer 2022-11-23T01:01:40.0664932Z 824caeed1492: Waiting 2022-11-23T01:01:40.0665103Z 6114b441ddbf: Pulling fs layer 2022-11-23T01:01:40.0665345Z 516276b2131d: Waiting 2022-11-23T01:01:40.0665526Z 52944c8173bb: Pulling fs layer 2022-11-23T01:01:40.0665732Z 1762e4b0a4dc: Waiting 2022-11-23T01:01:40.0665907Z 3507de6a25b1: Waiting 2022-11-23T01:01:40.0666089Z 1c1aab3b72ce: Pulling fs layer 2022-11-23T01:01:40.0666262Z 49880d42ecce: Waiting 2022-11-23T01:01:40.0666449Z 9af1bdb6cccf: Pulling fs layer 2022-11-23T01:01:40.0666635Z 093e8e8bcad0: Waiting 2022-11-23T01:01:40.0666842Z 4694407ca3de: Waiting 2022-11-23T01:01:40.0667032Z b2c751e637f5: Waiting 2022-11-23T01:01:40.0667213Z 2b8671dbd244: Pulling fs layer 2022-11-23T01:01:40.0667379Z 6114b441ddbf: Waiting 2022-11-23T01:01:40.0667562Z 0ccc0a3f9f55: Pulling fs layer 2022-11-23T01:01:40.0667745Z 52944c8173bb: Waiting 2022-11-23T01:01:40.0667911Z 6c09ee684801: Pulling fs layer 2022-11-23T01:01:40.0668193Z 2b8671dbd244: Waiting 2022-11-23T01:01:40.0668381Z dde59f92df16: Pulling fs layer 2022-11-23T01:01:40.0668730Z 7108366bb357: Pulling fs layer 2022-11-23T01:01:40.0668961Z a17d3f0d245f: Pulling fs layer 2022-11-23T01:01:40.0669149Z 1c1aab3b72ce: Waiting 2022-11-23T01:01:40.0669343Z af1f57b03dc1: Pulling fs layer 2022-11-23T01:01:40.0669523Z dde59f92df16: Waiting 2022-11-23T01:01:40.0669713Z fedc8844da0c: Pulling fs layer 2022-11-23T01:01:40.0669900Z a17d3f0d245f: Waiting 2022-11-23T01:01:40.0670070Z 4d9ee4a84202: Pulling fs layer 2022-11-23T01:01:40.0670257Z af1f57b03dc1: Waiting 2022-11-23T01:01:40.0670437Z 422bb402971f: Pulling fs layer 2022-11-23T01:01:40.0670616Z 30d9fd19fcf8: Pulling fs layer 2022-11-23T01:01:40.0670798Z 4d9ee4a84202: Waiting 2022-11-23T01:01:40.0670967Z 422bb402971f: Waiting 2022-11-23T01:01:40.0671135Z e6ec9ff72352: Pulling fs layer 2022-11-23T01:01:40.0671329Z b2dabb47a4a1: Pulling fs layer 2022-11-23T01:01:40.0671524Z 78592748cf6a: Pulling fs layer 2022-11-23T01:01:40.0671700Z 2f62c284cd73: Pulling fs layer 2022-11-23T01:01:40.0671893Z d25a76b3a031: Pulling fs layer 2022-11-23T01:01:40.0672149Z 5a50d7c62a26: Pulling fs layer 2022-11-23T01:01:40.0672334Z cd6d94bb12b1: Pulling fs layer 2022-11-23T01:01:40.0672530Z 2fba50966c3e: Pulling fs layer 2022-11-23T01:01:40.0672716Z 2f62c284cd73: Waiting 2022-11-23T01:01:40.0672882Z d25a76b3a031: Waiting 2022-11-23T01:01:40.0673055Z 5a50d7c62a26: Waiting 2022-11-23T01:01:40.1438675Z 96afc5db69dc: Verifying Checksum 2022-11-23T01:01:40.1438918Z 96afc5db69dc: Download complete 2022-11-23T01:01:40.2261703Z ed66d74b7c20: Verifying Checksum 2022-11-23T01:01:40.2262141Z ed66d74b7c20: Download complete 2022-11-23T01:01:40.3005805Z 42439e1d77c4: Verifying Checksum 2022-11-23T01:01:40.3006293Z 42439e1d77c4: Download complete 2022-11-23T01:01:40.3746762Z 80bf4ab5d594: Verifying Checksum 2022-11-23T01:01:40.3747199Z 80bf4ab5d594: Download complete 2022-11-23T01:01:40.3973974Z eaead16dc43b: Download complete 2022-11-23T01:01:40.4594700Z 879beb9f8e9b: Download complete 2022-11-23T01:01:40.4800313Z 3f623e97839a: Verifying Checksum 2022-11-23T01:01:40.4800789Z 3f623e97839a: Download complete 2022-11-23T01:01:40.5363989Z d845eb9c30a0: Verifying Checksum 2022-11-23T01:01:40.5364264Z d845eb9c30a0: Download complete 2022-11-23T01:01:40.5573681Z 8654f94b3130: Verifying Checksum 2022-11-23T01:01:40.5574328Z 8654f94b3130: Download complete 2022-11-23T01:01:40.6322812Z 0c91414faf6f: Download complete 2022-11-23T01:01:40.7269955Z 16b14aea0152: Verifying Checksum 2022-11-23T01:01:40.7270357Z 16b14aea0152: Download complete 2022-11-23T01:01:41.1195183Z eaead16dc43b: Pull complete 2022-11-23T01:01:41.2363018Z 96afc5db69dc: Pull complete 2022-11-23T01:01:41.4108960Z 0c488367249b: Verifying Checksum 2022-11-23T01:01:41.4109335Z 0c488367249b: Download complete 2022-11-23T01:01:41.4898499Z d703d3396c78: Download complete 2022-11-23T01:01:41.5654581Z acf65ded4ba0: Download complete 2022-11-23T01:01:42.0522203Z 19d7da72e211: Verifying Checksum 2022-11-23T01:01:42.0522479Z 19d7da72e211: Download complete 2022-11-23T01:01:42.1288622Z cb4b60b16a23: Verifying Checksum 2022-11-23T01:01:42.1288995Z cb4b60b16a23: Download complete 2022-11-23T01:01:42.2635643Z 6325a464626e: Verifying Checksum 2022-11-23T01:01:42.2636137Z 6325a464626e: Download complete 2022-11-23T01:01:42.3343290Z b17aded2bf76: Verifying Checksum 2022-11-23T01:01:42.3343685Z b17aded2bf76: Download complete 2022-11-23T01:01:42.4068445Z 79f0454f85ca: Download complete 2022-11-23T01:01:42.4708376Z 5df784aed3a4: Verifying Checksum 2022-11-23T01:01:42.4708693Z 5df784aed3a4: Download complete 2022-11-23T01:01:42.5723671Z 196c72536fc8: Verifying Checksum 2022-11-23T01:01:42.5724053Z 196c72536fc8: Download complete 2022-11-23T01:01:42.6454278Z e7cf72049767: Verifying Checksum 2022-11-23T01:01:42.6454525Z e7cf72049767: Download complete 2022-11-23T01:01:43.2102217Z 9cd06b79192a: Download complete 2022-11-23T01:01:43.2827013Z c62a223ffc22: Verifying Checksum 2022-11-23T01:01:43.2827435Z c62a223ffc22: Download complete 2022-11-23T01:01:43.3340986Z 824caeed1492: Verifying Checksum 2022-11-23T01:01:43.3341481Z 824caeed1492: Download complete 2022-11-23T01:01:43.4128633Z 1762e4b0a4dc: Verifying Checksum 2022-11-23T01:01:43.4129085Z 1762e4b0a4dc: Download complete 2022-11-23T01:01:43.4905410Z 093e8e8bcad0: Download complete 2022-11-23T01:01:43.5707257Z 133752b42076: Verifying Checksum 2022-11-23T01:01:43.5707672Z 133752b42076: Download complete 2022-11-23T01:01:43.5767475Z 7c4667efa9a5: Verifying Checksum 2022-11-23T01:01:43.5767866Z 7c4667efa9a5: Download complete 2022-11-23T01:01:43.6346656Z 3507de6a25b1: Download complete 2022-11-23T01:01:43.6951309Z 49880d42ecce: Verifying Checksum 2022-11-23T01:01:43.6951735Z 49880d42ecce: Download complete 2022-11-23T01:01:43.7675152Z 4694407ca3de: Download complete 2022-11-23T01:01:43.8423461Z b2c751e637f5: Verifying Checksum 2022-11-23T01:01:43.8423863Z b2c751e637f5: Download complete 2022-11-23T01:01:43.9360401Z 6114b441ddbf: Verifying Checksum 2022-11-23T01:01:43.9360920Z 6114b441ddbf: Download complete 2022-11-23T01:01:44.0273581Z 52944c8173bb: Verifying Checksum 2022-11-23T01:01:44.0275003Z 52944c8173bb: Download complete 2022-11-23T01:01:44.1022269Z 1c1aab3b72ce: Download complete 2022-11-23T01:01:44.2073289Z 9af1bdb6cccf: Download complete 2022-11-23T01:01:44.2760120Z 2b8671dbd244: Verifying Checksum 2022-11-23T01:01:44.2760566Z 2b8671dbd244: Download complete 2022-11-23T01:01:44.3474537Z 0ccc0a3f9f55: Verifying Checksum 2022-11-23T01:01:44.3474904Z 0ccc0a3f9f55: Download complete 2022-11-23T01:01:44.4164521Z 6c09ee684801: Verifying Checksum 2022-11-23T01:01:44.4164978Z 6c09ee684801: Download complete 2022-11-23T01:01:44.4842274Z dde59f92df16: Verifying Checksum 2022-11-23T01:01:44.4843061Z dde59f92df16: Download complete 2022-11-23T01:01:44.9539607Z 7108366bb357: Download complete 2022-11-23T01:01:45.0325294Z a17d3f0d245f: Verifying Checksum 2022-11-23T01:01:45.0325708Z a17d3f0d245f: Download complete 2022-11-23T01:01:45.1372927Z af1f57b03dc1: Verifying Checksum 2022-11-23T01:01:45.1373374Z af1f57b03dc1: Download complete 2022-11-23T01:01:45.2161016Z fedc8844da0c: Verifying Checksum 2022-11-23T01:01:45.2162204Z fedc8844da0c: Download complete 2022-11-23T01:01:45.2998690Z 4d9ee4a84202: Verifying Checksum 2022-11-23T01:01:45.2999472Z 4d9ee4a84202: Download complete 2022-11-23T01:01:45.5566093Z 422bb402971f: Verifying Checksum 2022-11-23T01:01:45.5567254Z 422bb402971f: Download complete 2022-11-23T01:01:45.6523052Z 30d9fd19fcf8: Verifying Checksum 2022-11-23T01:01:45.6523889Z 30d9fd19fcf8: Download complete 2022-11-23T01:01:45.7245787Z e6ec9ff72352: Download complete 2022-11-23T01:01:45.8183249Z 516276b2131d: Verifying Checksum 2022-11-23T01:01:45.8183896Z 516276b2131d: Download complete 2022-11-23T01:01:45.8944743Z 78592748cf6a: Download complete 2022-11-23T01:01:45.9889307Z 2f62c284cd73: Download complete 2022-11-23T01:01:46.0719062Z d25a76b3a031: Download complete 2022-11-23T01:01:46.1361510Z 5a50d7c62a26: Download complete 2022-11-23T01:01:46.2001896Z cd6d94bb12b1: Download complete 2022-11-23T01:01:46.8015096Z 2fba50966c3e: Verifying Checksum 2022-11-23T01:01:46.8015524Z 2fba50966c3e: Download complete 2022-11-23T01:01:49.8824536Z 9cd06b79192a: Pull complete 2022-11-23T01:01:50.0945757Z ed66d74b7c20: Pull complete 2022-11-23T01:01:50.2562535Z 42439e1d77c4: Pull complete 2022-11-23T01:01:50.3785988Z b2dabb47a4a1: Verifying Checksum 2022-11-23T01:01:50.3786305Z b2dabb47a4a1: Download complete 2022-11-23T01:01:50.4876047Z 80bf4ab5d594: Pull complete 2022-11-23T01:01:50.7299411Z 879beb9f8e9b: Pull complete 2022-11-23T01:01:50.9891968Z 3f623e97839a: Pull complete 2022-11-23T01:01:51.2371996Z d845eb9c30a0: Pull complete 2022-11-23T01:01:51.4773613Z 8654f94b3130: Pull complete 2022-11-23T01:01:53.3393052Z a2871863f729: Verifying Checksum 2022-11-23T01:01:53.5881794Z 0c488367249b: Pull complete 2022-11-23T01:01:54.0413467Z 0c91414faf6f: Pull complete 2022-11-23T01:01:54.3609313Z 16b14aea0152: Pull complete 2022-11-23T01:02:16.7548487Z a2871863f729: Pull complete 2022-11-23T01:02:17.1955346Z d703d3396c78: Pull complete 2022-11-23T01:02:17.4429023Z acf65ded4ba0: Pull complete 2022-11-23T01:02:18.2415245Z 19d7da72e211: Pull complete 2022-11-23T01:02:18.4220627Z cb4b60b16a23: Pull complete 2022-11-23T01:02:18.6086784Z 6325a464626e: Pull complete 2022-11-23T01:02:18.8172393Z b17aded2bf76: Pull complete 2022-11-23T01:02:18.9536767Z 79f0454f85ca: Pull complete 2022-11-23T01:02:19.1407767Z 5df784aed3a4: Pull complete 2022-11-23T01:02:19.3645521Z 196c72536fc8: Pull complete 2022-11-23T01:02:19.5938705Z e7cf72049767: Pull complete 2022-11-23T01:02:21.5962442Z 7c4667efa9a5: Pull complete 2022-11-23T01:02:21.8324482Z c62a223ffc22: Pull complete 2022-11-23T01:02:21.9954318Z 824caeed1492: Pull complete 2022-11-23T01:02:22.1333239Z 1762e4b0a4dc: Pull complete 2022-11-23T01:02:22.2233063Z 093e8e8bcad0: Pull complete 2022-11-23T01:02:22.3150474Z 133752b42076: Pull complete 2022-11-23T01:02:26.9021059Z 516276b2131d: Pull complete 2022-11-23T01:02:26.9999543Z 3507de6a25b1: Pull complete 2022-11-23T01:02:27.0949663Z 49880d42ecce: Pull complete 2022-11-23T01:02:27.1997288Z 4694407ca3de: Pull complete 2022-11-23T01:02:27.3131936Z b2c751e637f5: Pull complete 2022-11-23T01:02:27.4078984Z 6114b441ddbf: Pull complete 2022-11-23T01:02:27.5039900Z 52944c8173bb: Pull complete 2022-11-23T01:02:27.6078880Z 1c1aab3b72ce: Pull complete 2022-11-23T01:02:27.6797425Z 9af1bdb6cccf: Pull complete 2022-11-23T01:02:27.7918708Z 2b8671dbd244: Pull complete 2022-11-23T01:02:27.8794885Z 0ccc0a3f9f55: Pull complete 2022-11-23T01:02:27.9686372Z 6c09ee684801: Pull complete 2022-11-23T01:02:28.0676394Z dde59f92df16: Pull complete 2022-11-23T01:02:29.3303356Z 7108366bb357: Pull complete 2022-11-23T01:02:29.5147812Z a17d3f0d245f: Pull complete 2022-11-23T01:02:29.6513369Z af1f57b03dc1: Pull complete 2022-11-23T01:02:29.9005044Z fedc8844da0c: Pull complete 2022-11-23T01:02:30.1324162Z 4d9ee4a84202: Pull complete 2022-11-23T01:02:30.6011013Z 422bb402971f: Pull complete 2022-11-23T01:02:30.8472608Z 30d9fd19fcf8: Pull complete 2022-11-23T01:02:31.0954835Z e6ec9ff72352: Pull complete 2022-11-23T01:02:35.7676813Z b2dabb47a4a1: Pull complete 2022-11-23T01:02:35.8629028Z 78592748cf6a: Pull complete 2022-11-23T01:02:35.9437469Z 2f62c284cd73: Pull complete 2022-11-23T01:02:36.0496923Z d25a76b3a031: Pull complete 2022-11-23T01:02:36.1350286Z 5a50d7c62a26: Pull complete 2022-11-23T01:02:36.2158522Z cd6d94bb12b1: Pull complete 2022-11-23T01:02:37.7923113Z 2fba50966c3e: Pull complete 2022-11-23T01:02:37.8021461Z Digest: sha256:128a8060bc8d0d2006ae18b574265c3aa935b731c43947b53188b825fa718d6b 2022-11-23T01:02:37.8045298Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940 2022-11-23T01:02:37.8072142Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940 2022-11-23T01:02:37.8127399Z ##[group]Run python3 -m pip install psutil==5.9.1 2022-11-23T01:02:37.8127710Z python3 -m pip install psutil==5.9.1 2022-11-23T01:02:37.8127960Z python3 -m pip install pynvml==11.4.1 2022-11-23T01:02:37.8128246Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2022-11-23T01:02:37.8128554Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2022-11-23T01:02:37.8139913Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:02:37.8140165Z env: 2022-11-23T01:02:37.8140370Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:02:37.8140572Z ##[endgroup] 2022-11-23T01:02:38.5628641Z Defaulting to user installation because normal site-packages is not writeable 2022-11-23T01:02:38.5845251Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1) 2022-11-23T01:02:39.0268345Z Defaulting to user installation because normal site-packages is not writeable 2022-11-23T01:02:39.0452452Z Requirement already satisfied: pynvml==11.4.1 in /home/ec2-user/.local/lib/python3.7/site-packages (11.4.1) 2022-11-23T01:02:39.2593794Z Prepare all required actions 2022-11-23T01:02:39.2594086Z Getting action download info 2022-11-23T01:02:39.4414197Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:4a8bfae15cc25cc0785c1603ee87a9da8fd442ea) 2022-11-23T01:02:39.6911546Z Download action repository 'actions/download-artifact@v3' (SHA:9782bd6a9848b53b110e712e20e42d89988822b7) 2022-11-23T01:02:39.8083733Z ##[group]Run ./.github/actions/download-build-artifacts 2022-11-23T01:02:39.8083947Z with: 2022-11-23T01:02:39.8084133Z name: linux-focal-py3.7-gcc7 2022-11-23T01:02:39.8084316Z env: 2022-11-23T01:02:39.8084474Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:02:39.8084659Z ##[endgroup] 2022-11-23T01:02:39.8106924Z ##[group]Run seemethere/download-artifact-s3@v4 2022-11-23T01:02:39.8107141Z with: 2022-11-23T01:02:39.8107314Z name: linux-focal-py3.7-gcc7 2022-11-23T01:02:39.8107521Z s3-bucket: gha-artifacts 2022-11-23T01:02:39.8107707Z region: us-east-1 2022-11-23T01:02:39.8107858Z env: 2022-11-23T01:02:39.8108060Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:02:39.8108233Z ##[endgroup] 2022-11-23T01:02:40.2806283Z Found 1 objects with prefix pytorch/pytorch/3528293562/linux-focal-py3.7-gcc7/ 2022-11-23T01:02:40.2806761Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2022-11-23T01:02:43.3804524Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2022-11-23T01:02:43.3804956Z 2022-11-23T01:02:43.3822789Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/ 2022-11-23T01:02:43.3832782Z Artifact download has finished successfully 2022-11-23T01:02:43.3933356Z ##[group]Run unzip -o artifacts.zip 2022-11-23T01:02:43.3933586Z unzip -o artifacts.zip 2022-11-23T01:02:43.3944866Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:02:43.3945084Z env: 2022-11-23T01:02:43.3945258Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:02:43.3945433Z ##[endgroup] 2022-11-23T01:02:43.4079650Z Archive: artifacts.zip 2022-11-23T01:02:43.4080855Z creating: dist/ 2022-11-23T01:02:44.1662434Z inflating: dist/torch-1.14.0a0+git1cfd385-cp37-cp37m-linux_x86_64.whl 2022-11-23T01:02:44.1662815Z creating: build/custom_test_artifacts/ 2022-11-23T01:02:44.1663343Z creating: build/custom_test_artifacts/custom-op-build/ 2022-11-23T01:02:44.1663684Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2022-11-23T01:02:44.1664743Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeOutput.log 2022-11-23T01:02:44.1665148Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/ 2022-11-23T01:02:44.1665566Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeSystem.cmake 2022-11-23T01:02:44.1665984Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/ 2022-11-23T01:02:44.1666400Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/tmp/ 2022-11-23T01:02:44.1668129Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c 2022-11-23T01:02:44.1669404Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/a.out 2022-11-23T01:02:44.1669811Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/ 2022-11-23T01:02:44.1670227Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/ 2022-11-23T01:02:44.1671915Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp 2022-11-23T01:02:44.1673300Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out 2022-11-23T01:02:44.1674729Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin 2022-11-23T01:02:44.1675200Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake 2022-11-23T01:02:44.1676574Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin 2022-11-23T01:02:44.1677424Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake 2022-11-23T01:02:44.1677910Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2022-11-23T01:02:44.1678352Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeError.log 2022-11-23T01:02:44.1678838Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2022-11-23T01:02:44.1679241Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2022-11-23T01:02:44.1679842Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2022-11-23T01:02:44.1680311Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2022-11-23T01:02:44.1680938Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2022-11-23T01:02:44.1681372Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2022-11-23T01:02:44.1681805Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2022-11-23T01:02:44.1682397Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2022-11-23T01:02:44.1682846Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2022-11-23T01:02:44.1683284Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2022-11-23T01:02:44.1683715Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2022-11-23T01:02:44.1699297Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2022-11-23T01:02:44.1787050Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2022-11-23T01:02:44.1787492Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2022-11-23T01:02:44.1788057Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2022-11-23T01:02:44.1788534Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2022-11-23T01:02:44.1788991Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2022-11-23T01:02:44.1789436Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2022-11-23T01:02:44.1789888Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2022-11-23T01:02:44.1790340Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2022-11-23T01:02:44.1790794Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2022-11-23T01:02:44.1791248Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2022-11-23T01:02:44.1791688Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2022-11-23T01:02:44.1807024Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2022-11-23T01:02:44.1870443Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2022-11-23T01:02:44.1870931Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2022-11-23T01:02:44.1871689Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2022-11-23T01:02:44.1872176Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2022-11-23T01:02:44.1872585Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2022-11-23T01:02:44.1872980Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2022-11-23T01:02:44.1874189Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2022-11-23T01:02:44.1874776Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2022-11-23T01:02:44.1875599Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2022-11-23T01:02:44.1946683Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2022-11-23T01:02:44.1995462Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2022-11-23T01:02:44.1995826Z creating: build/custom_test_artifacts/jit-hook-build/ 2022-11-23T01:02:44.1996231Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2022-11-23T01:02:44.1998866Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeOutput.log 2022-11-23T01:02:44.1999391Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/ 2022-11-23T01:02:44.1999896Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeSystem.cmake 2022-11-23T01:02:44.2000312Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/ 2022-11-23T01:02:44.2000868Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/tmp/ 2022-11-23T01:02:44.2001921Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c 2022-11-23T01:02:44.2003356Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/a.out 2022-11-23T01:02:44.2003815Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/ 2022-11-23T01:02:44.2004234Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/ 2022-11-23T01:02:44.2005667Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp 2022-11-23T01:02:44.2006935Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out 2022-11-23T01:02:44.2008253Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin 2022-11-23T01:02:44.2008835Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake 2022-11-23T01:02:44.2010211Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin 2022-11-23T01:02:44.2010987Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake 2022-11-23T01:02:44.2011522Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2022-11-23T01:02:44.2011912Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeError.log 2022-11-23T01:02:44.2012327Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2022-11-23T01:02:44.2012729Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2022-11-23T01:02:44.2013162Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2022-11-23T01:02:44.2013631Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2022-11-23T01:02:44.2014088Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2022-11-23T01:02:44.2014519Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2022-11-23T01:02:44.2014949Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2022-11-23T01:02:44.2015518Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2022-11-23T01:02:44.2015983Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2022-11-23T01:02:44.2016429Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2022-11-23T01:02:44.2017021Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2022-11-23T01:02:44.2031679Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2022-11-23T01:02:44.2080316Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2022-11-23T01:02:44.2081010Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2022-11-23T01:02:44.2081462Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2022-11-23T01:02:44.2081885Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2022-11-23T01:02:44.2082287Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2022-11-23T01:02:44.2082817Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2022-11-23T01:02:44.2084157Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2022-11-23T01:02:44.2084944Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2022-11-23T01:02:44.2085373Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2022-11-23T01:02:44.2123090Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2022-11-23T01:02:44.2123458Z creating: build/custom_test_artifacts/custom-backend-build/ 2022-11-23T01:02:44.2123832Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2022-11-23T01:02:44.2126041Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeOutput.log 2022-11-23T01:02:44.2126465Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/ 2022-11-23T01:02:44.2126903Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeSystem.cmake 2022-11-23T01:02:44.2127347Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/ 2022-11-23T01:02:44.2127885Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/tmp/ 2022-11-23T01:02:44.2129117Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c 2022-11-23T01:02:44.2130416Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/a.out 2022-11-23T01:02:44.2130861Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/ 2022-11-23T01:02:44.2131565Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/ 2022-11-23T01:02:44.2133327Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp 2022-11-23T01:02:44.2134590Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out 2022-11-23T01:02:44.2135977Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin 2022-11-23T01:02:44.2136462Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake 2022-11-23T01:02:44.2137850Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin 2022-11-23T01:02:44.2138491Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake 2022-11-23T01:02:44.2138932Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2022-11-23T01:02:44.2139422Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeError.log 2022-11-23T01:02:44.2139861Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2022-11-23T01:02:44.2140279Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2022-11-23T01:02:44.2140754Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2022-11-23T01:02:44.2141254Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2022-11-23T01:02:44.2141744Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2022-11-23T01:02:44.2142200Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2022-11-23T01:02:44.2142675Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2022-11-23T01:02:44.2143147Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2022-11-23T01:02:44.2143625Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2022-11-23T01:02:44.2144083Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2022-11-23T01:02:44.2144558Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2022-11-23T01:02:44.2147139Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2022-11-23T01:02:44.2260552Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2022-11-23T01:02:44.2261019Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2022-11-23T01:02:44.2261511Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2022-11-23T01:02:44.2262024Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2022-11-23T01:02:44.2262515Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2022-11-23T01:02:44.2262981Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2022-11-23T01:02:44.2263558Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2022-11-23T01:02:44.2264381Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2022-11-23T01:02:44.2264951Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2022-11-23T01:02:44.2266104Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2022-11-23T01:02:44.2267051Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2022-11-23T01:02:44.2283302Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2022-11-23T01:02:44.2328601Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2022-11-23T01:02:44.2329580Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2022-11-23T01:02:44.2330453Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2022-11-23T01:02:44.2331272Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2022-11-23T01:02:44.2332022Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2022-11-23T01:02:44.2332931Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2022-11-23T01:02:44.2333691Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2022-11-23T01:02:44.2334618Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2022-11-23T01:02:44.2335342Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2022-11-23T01:02:44.2427611Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2022-11-23T01:02:44.2463709Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2022-11-23T01:02:44.2464192Z creating: build/lib/ 2022-11-23T01:02:44.2464585Z inflating: build/lib/libclog.a 2022-11-23T01:02:44.2516048Z inflating: build/lib/libgtest.a 2022-11-23T01:02:44.2524376Z inflating: build/lib/libpthreadpool.a 2022-11-23T01:02:44.2596486Z inflating: build/lib/libbenchmark.a 2022-11-23T01:02:44.2603596Z inflating: build/lib/libittnotify.a 2022-11-23T01:02:44.2686285Z inflating: build/lib/libprotobuf-lite.a 2022-11-23T01:02:44.2711210Z inflating: build/lib/libtensorpipe_uv.a 2022-11-23T01:02:44.2801843Z inflating: build/lib/libgloo.a 2022-11-23T01:02:44.2861938Z inflating: build/lib/libasmjit.a 2022-11-23T01:02:44.3277978Z inflating: build/lib/libprotobuf.a 2022-11-23T01:02:44.3303454Z inflating: build/lib/libfmt.a 2022-11-23T01:02:44.3304018Z inflating: build/lib/libfoxi_loader.a 2022-11-23T01:02:44.3366730Z inflating: build/lib/libc10.so 2022-11-23T01:02:44.3367896Z inflating: build/lib/libtorch_global_deps.so 2022-11-23T01:02:44.3375872Z inflating: build/lib/libcpuinfo.a 2022-11-23T01:02:44.3383440Z inflating: build/lib/libcpuinfo_internals.a 2022-11-23T01:02:44.3396205Z inflating: build/lib/libqnnpack.a 2022-11-23T01:02:44.3398556Z inflating: build/lib/libnnpack_reference_layers.a 2022-11-23T01:02:44.3843528Z inflating: build/lib/libprotoc.a 2022-11-23T01:02:44.3858751Z inflating: build/lib/libgmock.a 2022-11-23T01:02:44.3877216Z inflating: build/lib/libpytorch_qnnpack.a 2022-11-23T01:02:44.3877931Z inflating: build/lib/libbenchmark_main.a 2022-11-23T01:02:44.3878659Z inflating: build/lib/libgtest_main.a 2022-11-23T01:02:45.1505971Z inflating: build/lib/libdnnl.a 2022-11-23T01:02:45.2016485Z inflating: build/lib/libtensorpipe.a 2022-11-23T01:02:45.2033828Z inflating: build/lib/libnnpack.a 2022-11-23T01:02:45.3228729Z inflating: build/lib/libfbgemm.a 2022-11-23T01:02:45.4120428Z inflating: build/lib/libdnnl_graph.a 2022-11-23T01:02:45.4121088Z inflating: build/lib/libgmock_main.a 2022-11-23T01:02:45.4351650Z inflating: build/lib/libkineto.a 2022-11-23T01:02:45.4463040Z inflating: build/lib/libXNNPACK.a 2022-11-23T01:02:45.4498668Z inflating: build/lib/libcaffe2_protos.a 2022-11-23T01:02:45.4536620Z inflating: build/lib/libonnx_proto.a 2022-11-23T01:02:45.5066034Z inflating: build/lib/libonnx.a 2022-11-23T01:02:47.3549314Z inflating: build/lib/libtorch_cpu.so 2022-11-23T01:02:47.3549892Z inflating: build/lib/libtorch.so 2022-11-23T01:02:47.3597481Z inflating: build/lib/libtorchbind_test.so 2022-11-23T01:02:47.3616380Z inflating: build/lib/libjitbackend_test.so 2022-11-23T01:02:47.3641324Z inflating: build/lib/libbackend_with_compiler.so 2022-11-23T01:02:47.3644603Z inflating: build/lib/libshm.so 2022-11-23T01:02:47.4968166Z inflating: build/lib/libtorch_python.so 2022-11-23T01:02:47.4998679Z inflating: build/lib/libnnapi_backend.so 2022-11-23T01:02:47.4999063Z creating: build/bin/ 2022-11-23T01:02:47.4999430Z creating: build/bin/CMakeFiles/ 2022-11-23T01:02:47.4999924Z inflating: build/bin/CMakeFiles/CMakeDirectoryInformation.cmake 2022-11-23T01:02:47.5000426Z extracting: build/bin/CMakeFiles/progress.marks 2022-11-23T01:02:47.5042778Z inflating: build/bin/weakref_test 2022-11-23T01:02:47.5043480Z inflating: build/bin/Makefile 2022-11-23T01:02:47.5045156Z inflating: build/bin/verify_api_visibility 2022-11-23T01:02:47.5045624Z inflating: build/bin/cmake_install.cmake 2022-11-23T01:02:47.5104106Z inflating: build/bin/vmap_test 2022-11-23T01:02:47.5104543Z inflating: build/bin/CTestTestfile.cmake 2022-11-23T01:02:47.5145439Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2022-11-23T01:02:47.5188787Z inflating: build/bin/c10_DeviceGuard_test 2022-11-23T01:02:47.5230971Z inflating: build/bin/c10_Device_test 2022-11-23T01:02:47.5280469Z inflating: build/bin/c10_DispatchKeySet_test 2022-11-23T01:02:47.5320772Z inflating: build/bin/c10_StreamGuard_test 2022-11-23T01:02:47.5361899Z inflating: build/bin/c10_SymInt_test 2022-11-23T01:02:47.5408968Z inflating: build/bin/c10_InlineDeviceGuard_test 2022-11-23T01:02:47.5455689Z inflating: build/bin/c10_InlineStreamGuard_test 2022-11-23T01:02:47.5496105Z inflating: build/bin/c10_Array_test 2022-11-23T01:02:47.5543926Z inflating: build/bin/c10_SizesAndStrides_test 2022-11-23T01:02:47.5588287Z inflating: build/bin/c10_Bitset_test 2022-11-23T01:02:47.5631221Z inflating: build/bin/c10_C++17_test 2022-11-23T01:02:47.5671368Z inflating: build/bin/c10_ConstexprCrc_test 2022-11-23T01:02:47.5712488Z inflating: build/bin/c10_DeadlockDetection_test 2022-11-23T01:02:47.5754212Z inflating: build/bin/c10_Half_test 2022-11-23T01:02:47.5801625Z inflating: build/bin/c10_LeftRight_test 2022-11-23T01:02:47.5854258Z inflating: build/bin/c10_Metaprogramming_test 2022-11-23T01:02:47.5975113Z inflating: build/bin/c10_SmallVectorTest 2022-11-23T01:02:47.6017175Z inflating: build/bin/c10_Synchronized_test 2022-11-23T01:02:47.6065122Z inflating: build/bin/c10_ThreadLocal_test 2022-11-23T01:02:47.6109000Z inflating: build/bin/c10_TypeIndex_test 2022-11-23T01:02:47.6151410Z inflating: build/bin/c10_TypeList_test 2022-11-23T01:02:47.6192365Z inflating: build/bin/c10_TypeTraits_test 2022-11-23T01:02:47.6235530Z inflating: build/bin/c10_accumulate_test 2022-11-23T01:02:47.6282840Z inflating: build/bin/c10_bfloat16_test 2022-11-23T01:02:47.6327998Z inflating: build/bin/c10_complex_math_test 2022-11-23T01:02:47.6374840Z inflating: build/bin/c10_complex_test 2022-11-23T01:02:47.6418780Z inflating: build/bin/c10_exception_test 2022-11-23T01:02:47.6510518Z inflating: build/bin/c10_either_test 2022-11-23T01:02:47.6552408Z inflating: build/bin/c10_flags_test 2022-11-23T01:02:47.6694557Z inflating: build/bin/c10_intrusive_ptr_test 2022-11-23T01:02:47.6736716Z inflating: build/bin/c10_irange_test 2022-11-23T01:02:47.6785244Z inflating: build/bin/c10_logging_test 2022-11-23T01:02:47.6848339Z inflating: build/bin/c10_optional_test 2022-11-23T01:02:47.6900643Z inflating: build/bin/c10_ordered_preserving_dict_test 2022-11-23T01:02:47.6946465Z inflating: build/bin/c10_registry_test 2022-11-23T01:02:47.6996069Z inflating: build/bin/c10_string_view_test 2022-11-23T01:02:47.7043560Z inflating: build/bin/c10_intrusive_ptr_benchmark 2022-11-23T01:02:47.7086423Z inflating: build/bin/c10_tempfile_test 2022-11-23T01:02:47.7133957Z inflating: build/bin/c10_typeid_test 2022-11-23T01:02:47.7545238Z inflating: build/bin/protoc-3.13.0.0 2022-11-23T01:02:47.7956151Z inflating: build/bin/protoc 2022-11-23T01:02:47.8201913Z inflating: build/bin/vec_test_all_types_DEFAULT 2022-11-23T01:02:47.8478924Z inflating: build/bin/vec_test_all_types_AVX2 2022-11-23T01:02:47.8529714Z inflating: build/bin/TCPStoreTest 2022-11-23T01:02:47.8574591Z inflating: build/bin/HashStoreTest 2022-11-23T01:02:47.8619570Z inflating: build/bin/FileStoreTest 2022-11-23T01:02:47.8622072Z inflating: build/bin/example_allreduce 2022-11-23T01:02:47.8677138Z inflating: build/bin/ProcessGroupGlooTest 2022-11-23T01:02:47.8721237Z inflating: build/bin/Dimname_test 2022-11-23T01:02:47.8782477Z inflating: build/bin/Dict_test 2022-11-23T01:02:47.8840201Z inflating: build/bin/static_runtime_bench 2022-11-23T01:02:47.8893585Z inflating: build/bin/MaybeOwned_test 2022-11-23T01:02:47.8941550Z inflating: build/bin/NamedTensor_test 2022-11-23T01:02:47.8991012Z inflating: build/bin/apply_utils_test 2022-11-23T01:02:47.9206820Z inflating: build/bin/static_runtime_test 2022-11-23T01:02:47.9256140Z inflating: build/bin/atest 2022-11-23T01:02:47.9307633Z inflating: build/bin/basic 2022-11-23T01:02:47.9352934Z inflating: build/bin/broadcast_test 2022-11-23T01:02:47.9402039Z inflating: build/bin/cpu_generator_test 2022-11-23T01:02:47.9445746Z inflating: build/bin/cpu_profiling_allocator_test 2022-11-23T01:02:47.9487570Z inflating: build/bin/dispatch_key_set_test 2022-11-23T01:02:47.9529117Z inflating: build/bin/dlconvertor_test 2022-11-23T01:02:47.9603620Z inflating: build/bin/cpu_rng_test 2022-11-23T01:02:47.9652252Z inflating: build/bin/extension_backend_test 2022-11-23T01:02:47.9698902Z inflating: build/bin/half_test 2022-11-23T01:02:47.9740214Z inflating: build/bin/lazy_tensor_test 2022-11-23T01:02:47.9820005Z inflating: build/bin/ivalue_test 2022-11-23T01:02:47.9864999Z inflating: build/bin/memory_format_test 2022-11-23T01:02:47.9910234Z inflating: build/bin/math_kernel_test 2022-11-23T01:02:47.9955186Z inflating: build/bin/memory_overlapping_test 2022-11-23T01:02:47.9997683Z inflating: build/bin/operator_name_test 2022-11-23T01:02:48.0041645Z inflating: build/bin/mobile_memory_cleanup 2022-11-23T01:02:48.0088773Z inflating: build/bin/native_test 2022-11-23T01:02:48.0130783Z inflating: build/bin/operators_test 2022-11-23T01:02:48.0174875Z inflating: build/bin/packedtensoraccessor_test 2022-11-23T01:02:48.0229710Z inflating: build/bin/pow_test 2022-11-23T01:02:48.0277797Z inflating: build/bin/quantized_test 2022-11-23T01:02:48.0320320Z inflating: build/bin/reportMemoryUsage_test 2022-11-23T01:02:48.0361834Z inflating: build/bin/reduce_ops_test 2022-11-23T01:02:48.0409298Z inflating: build/bin/scalar_tensor_test 2022-11-23T01:02:48.0457641Z inflating: build/bin/scalar_test 2022-11-23T01:02:48.0501600Z inflating: build/bin/stride_properties_test 2022-11-23T01:02:48.0567859Z inflating: build/bin/tensor_iterator_test 2022-11-23T01:02:48.0569652Z inflating: build/bin/thread_init_test 2022-11-23T01:02:48.0616519Z inflating: build/bin/type_ptr_test 2022-11-23T01:02:48.0664202Z inflating: build/bin/test_parallel 2022-11-23T01:02:48.0705752Z inflating: build/bin/variant_test 2022-11-23T01:02:48.0756695Z inflating: build/bin/type_test 2022-11-23T01:02:48.0800447Z inflating: build/bin/undefined_tensor_test 2022-11-23T01:02:48.0843350Z inflating: build/bin/wrapdim_test 2022-11-23T01:02:48.0893637Z inflating: build/bin/IListRef_test 2022-11-23T01:02:48.0934551Z inflating: build/bin/xla_tensor_test 2022-11-23T01:02:48.1026544Z inflating: build/bin/List_test 2022-11-23T01:02:48.1129219Z inflating: build/bin/kernel_function_legacy_test 2022-11-23T01:02:48.1210233Z inflating: build/bin/kernel_function_test 2022-11-23T01:02:48.1265278Z inflating: build/bin/KernelFunction_test 2022-11-23T01:02:48.1373377Z inflating: build/bin/kernel_lambda_legacy_test 2022-11-23T01:02:48.1460477Z inflating: build/bin/kernel_lambda_test 2022-11-23T01:02:48.1510827Z inflating: build/bin/kernel_stackbased_test 2022-11-23T01:02:48.1591690Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2022-11-23T01:02:48.1631913Z inflating: build/bin/op_allowlist_test 2022-11-23T01:02:48.1674100Z inflating: build/bin/CppSignature_test 2022-11-23T01:02:48.1718489Z inflating: build/bin/inline_container_test 2022-11-23T01:02:48.1962281Z inflating: build/bin/op_registration_test 2022-11-23T01:02:48.2009597Z inflating: build/bin/backend_fallback_test 2022-11-23T01:02:48.2023438Z inflating: build/bin/tutorial_tensorexpr 2022-11-23T01:02:48.2068601Z inflating: build/bin/test_dist_autograd 2022-11-23T01:02:48.2127885Z inflating: build/bin/test_cpp_rpc 2022-11-23T01:02:48.2129827Z inflating: build/bin/parallel_benchmark 2022-11-23T01:02:48.2814151Z inflating: build/bin/test_tensorexpr 2022-11-23T01:02:48.2872235Z inflating: build/bin/test_mobile_nnc 2022-11-23T01:02:48.2881221Z inflating: build/bin/aot_model_compiler_test 2022-11-23T01:02:48.3180885Z inflating: build/bin/test_lazy 2022-11-23T01:02:48.3185222Z inflating: build/bin/torch_shm_manager 2022-11-23T01:02:48.4203234Z inflating: build/bin/test_api 2022-11-23T01:02:48.4699033Z inflating: build/bin/test_jit 2022-11-23T01:02:48.4700449Z inflating: .pytorch-test-times.json 2022-11-23T01:02:48.4722203Z ##[group]Run df -H 2022-11-23T01:02:48.4722387Z df -H 2022-11-23T01:02:48.4733515Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:02:48.4733717Z env: 2022-11-23T01:02:48.4733891Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:02:48.4734078Z ##[endgroup] 2022-11-23T01:02:48.4785872Z Filesystem Size Used Avail Use% Mounted on 2022-11-23T01:02:48.4786282Z devtmpfs 8.2G 0 8.2G 0% /dev 2022-11-23T01:02:48.4786586Z tmpfs 8.2G 8.3M 8.2G 1% /dev/shm 2022-11-23T01:02:48.4786866Z tmpfs 8.2G 431k 8.2G 1% /run 2022-11-23T01:02:48.4787197Z tmpfs 8.2G 0 8.2G 0% /sys/fs/cgroup 2022-11-23T01:02:48.4787503Z /dev/nvme0n1p1 162G 16G 146G 10% / 2022-11-23T01:02:48.4804620Z ##[group]Run .github/scripts/parse_ref.py 2022-11-23T01:02:48.4804848Z .github/scripts/parse_ref.py 2022-11-23T01:02:48.4815447Z shell: /usr/bin/bash -e {0} 2022-11-23T01:02:48.4815633Z env: 2022-11-23T01:02:48.4815794Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:02:48.4815985Z ##[endgroup] 2022-11-23T01:02:48.5032627Z ##[group]Run set -x 2022-11-23T01:02:48.5032936Z set -x 2022-11-23T01:02:48.5033128Z  2022-11-23T01:02:48.5033342Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2022-11-23T01:02:48.5033632Z  TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh 2022-11-23T01:02:48.5033916Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2022-11-23T01:02:48.5034168Z  TEST_COMMAND=.jenkins/caffe2/test.sh 2022-11-23T01:02:48.5034394Z else 2022-11-23T01:02:48.5034625Z  TEST_COMMAND=.jenkins/pytorch/test.sh 2022-11-23T01:02:48.5034851Z fi 2022-11-23T01:02:48.5035095Z  2022-11-23T01:02:48.5035358Z COMMIT_MESSAGES=$(git cherry -v "origin/${GIT_DEFAULT_BRANCH:-master}") 2022-11-23T01:02:48.5035621Z  2022-11-23T01:02:48.5035853Z # sanitize the input commit message and PR body here: 2022-11-23T01:02:48.5036094Z # 2022-11-23T01:02:48.5036404Z # trim all new lines from commit messages + PR_BODY to avoid issues with batch environment 2022-11-23T01:02:48.5036800Z # variable copying. see https://github.com/pytorch/pytorch/pull/80043#issuecomment-1167796028 2022-11-23T01:02:48.5037142Z COMMIT_MESSAGES="${COMMIT_MESSAGES//[$'\n\r']}" 2022-11-23T01:02:48.5037400Z PR_BODY="${PR_BODY//[$'\n\r']}" 2022-11-23T01:02:48.5037601Z  2022-11-23T01:02:48.5037889Z # then trim all special characters like single and double quotes to avoid unescaped inputs to 2022-11-23T01:02:48.5038195Z # wreak havoc internally 2022-11-23T01:02:48.5038458Z export COMMIT_MESSAGES="${COMMIT_MESSAGES//[\'\"]}" 2022-11-23T01:02:48.5038716Z export PR_BODY="${PR_BODY//[\'\"]}" 2022-11-23T01:02:48.5038931Z  2022-11-23T01:02:48.5039185Z # detached container should get cleaned up by teardown_ec2_linux 2022-11-23T01:02:48.5039498Z # TODO: Stop building test binaries as part of the build phase 2022-11-23T01:02:48.5039801Z # Used for GPU_FLAG since that doesn't play nice 2022-11-23T01:02:48.5040071Z # shellcheck disable=SC2086,SC2090 2022-11-23T01:02:48.5040318Z container_name=$(docker run \ 2022-11-23T01:02:48.5040535Z  ${GPU_FLAG:-} \ 2022-11-23T01:02:48.5040991Z  -e BUILD_ENVIRONMENT \ 2022-11-23T01:02:48.5041219Z  -e PR_NUMBER \ 2022-11-23T01:02:48.5041429Z  -e GITHUB_ACTIONS \ 2022-11-23T01:02:48.5041646Z  -e BASE_SHA \ 2022-11-23T01:02:48.5041857Z  -e BRANCH \ 2022-11-23T01:02:48.5042049Z  -e SHA1 \ 2022-11-23T01:02:48.5042269Z  -e AWS_DEFAULT_REGION \ 2022-11-23T01:02:48.5042499Z  -e IN_WHEEL_TEST \ 2022-11-23T01:02:48.5042706Z  -e SHARD_NUMBER \ 2022-11-23T01:02:48.5042924Z  -e TEST_CONFIG \ 2022-11-23T01:02:48.5043146Z  -e NUM_TEST_SHARDS \ 2022-11-23T01:02:48.5043352Z  -e PR_BODY \ 2022-11-23T01:02:48.5043572Z  -e COMMIT_MESSAGES \ 2022-11-23T01:02:48.5043813Z  -e PYTORCH_RETRY_TEST_CASES \ 2022-11-23T01:02:48.5044059Z  -e PYTORCH_OVERRIDE_FLAKY_SIGNAL \ 2022-11-23T01:02:48.5044296Z  -e PR_LABELS \ 2022-11-23T01:02:48.5044539Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2022-11-23T01:02:48.5044780Z  -e SCCACHE_BUCKET \ 2022-11-23T01:02:48.5044998Z  -e SCCACHE_S3_KEY_PREFIX \ 2022-11-23T01:02:48.5045223Z  -e XLA_CUDA \ 2022-11-23T01:02:48.5045460Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2022-11-23T01:02:48.5045710Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2022-11-23T01:02:48.5045982Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2022-11-23T01:02:48.5046269Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2022-11-23T01:02:48.5046522Z  --ulimit stack=10485760:83886080 \ 2022-11-23T01:02:48.5046781Z  --security-opt seccomp=unconfined \ 2022-11-23T01:02:48.5047112Z  --cap-add=SYS_PTRACE \ 2022-11-23T01:02:48.5047338Z  --ipc=host \ 2022-11-23T01:02:48.5047550Z  --shm-size="${SHM_SIZE}" \ 2022-11-23T01:02:48.5047767Z  --tty \ 2022-11-23T01:02:48.5047970Z  --detach \ 2022-11-23T01:02:48.5048184Z  --name="${container_name}" \ 2022-11-23T01:02:48.5048414Z  --user jenkins \ 2022-11-23T01:02:48.5048682Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2022-11-23T01:02:48.5048947Z  -w /var/lib/jenkins/workspace \ 2022-11-23T01:02:48.5049182Z  "${DOCKER_IMAGE}" 2022-11-23T01:02:48.5049383Z ) 2022-11-23T01:02:48.5049670Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2022-11-23T01:02:48.5050028Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2022-11-23T01:02:48.5060818Z shell: /usr/bin/bash -e {0} 2022-11-23T01:02:48.5061019Z env: 2022-11-23T01:02:48.5061225Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:02:48.5061487Z BUILD_ENVIRONMENT: linux-focal-py3.7-gcc7 2022-11-23T01:02:48.5061731Z PR_NUMBER: 2022-11-23T01:02:48.5061918Z BRANCH: master 2022-11-23T01:02:48.5062148Z SHA1: 1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T01:02:48.5062411Z BASE_SHA: 1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T01:02:48.5062646Z PYTORCH_RETRY_TEST_CASES: 1 2022-11-23T01:02:48.5062885Z PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1 2022-11-23T01:02:48.5063120Z TEST_CONFIG: distributed 2022-11-23T01:02:48.5063321Z SHARD_NUMBER: 2 2022-11-23T01:02:48.5063525Z NUM_TEST_SHARDS: 2 2022-11-23T01:02:48.5063729Z PR_BODY: 2022-11-23T01:02:48.5063971Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-11-23T01:02:48.5064243Z SCCACHE_S3_KEY_PREFIX: pull 2022-11-23T01:02:48.5064461Z SHM_SIZE: 1g 2022-11-23T01:02:48.5064814Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940 2022-11-23T01:02:48.5065165Z XLA_CUDA: 2022-11-23T01:02:48.5065454Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-11-23T01:02:48.5065764Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2022-11-23T01:02:48.5066001Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2022-11-23T01:02:48.5066229Z ##[endgroup] 2022-11-23T01:02:48.5090951Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2022-11-23T01:02:48.5091521Z + [[ linux-focal-py3.7-gcc7 == *onnx* ]] 2022-11-23T01:02:48.5091871Z + TEST_COMMAND=.jenkins/pytorch/test.sh 2022-11-23T01:02:48.5093953Z ++ git cherry -v origin/master 2022-11-23T01:02:48.5107709Z + COMMIT_MESSAGES= 2022-11-23T01:02:48.5108230Z + COMMIT_MESSAGES= 2022-11-23T01:02:48.5108519Z + PR_BODY= 2022-11-23T01:02:48.5108952Z + export COMMIT_MESSAGES= 2022-11-23T01:02:48.5109303Z + COMMIT_MESSAGES= 2022-11-23T01:02:48.5109622Z + export PR_BODY= 2022-11-23T01:02:48.5109907Z + PR_BODY= 2022-11-23T01:02:48.5117497Z +++ nproc --ignore=2 2022-11-23T01:02:48.5128621Z ++ docker run -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e COMMIT_MESSAGES -e PYTORCH_RETRY_TEST_CASES -e PYTORCH_OVERRIDE_FLAKY_SIGNAL -e PR_LABELS -e MAX_JOBS=6 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS --env-file=/tmp/github_env_3528293562 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=1g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940 2022-11-23T01:02:59.2727688Z + container_name=6e200e7a94c0c354a3a0a6fc7e5a55f6d42c9daee3c19a19715f86f8daa96d25 2022-11-23T01:02:59.2728320Z + echo DOCKER_CONTAINER_ID=6e200e7a94c0c354a3a0a6fc7e5a55f6d42c9daee3c19a19715f86f8daa96d25 2022-11-23T01:02:59.2731673Z ++ echo dist/torch-1.14.0a0+git1cfd385-cp37-cp37m-linux_x86_64.whl 2022-11-23T01:02:59.2733240Z + docker exec -t 6e200e7a94c0c354a3a0a6fc7e5a55f6d42c9daee3c19a19715f86f8daa96d25 sh -c 'pip install dist/torch-1.14.0a0+git1cfd385-cp37-cp37m-linux_x86_64.whl[opt-einsum] && .jenkins/pytorch/test.sh' 2022-11-23T01:02:59.6794072Z Processing ./dist/torch-1.14.0a0+git1cfd385-cp37-cp37m-linux_x86_64.whl 2022-11-23T01:03:00.0415860Z Requirement already satisfied: sympy in /opt/conda/lib/python3.7/site-packages (from torch==1.14.0a0+git1cfd385) (1.10.1) 2022-11-23T01:03:00.0418661Z Requirement already satisfied: networkx in /opt/conda/lib/python3.7/site-packages (from torch==1.14.0a0+git1cfd385) (2.6.3) 2022-11-23T01:03:00.0421420Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.7/site-packages (from torch==1.14.0a0+git1cfd385) (4.3.0) 2022-11-23T01:03:00.0434935Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/lib/python3.7/site-packages (from torch==1.14.0a0+git1cfd385) (3.3.0) 2022-11-23T01:03:00.0500594Z Requirement already satisfied: numpy>=1.7 in /opt/conda/lib/python3.7/site-packages (from opt-einsum>=3.3->torch==1.14.0a0+git1cfd385) (1.18.5) 2022-11-23T01:03:00.0683205Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.7/site-packages (from sympy->torch==1.14.0a0+git1cfd385) (1.2.1) 2022-11-23T01:03:00.8384105Z Installing collected packages: torch 2022-11-23T01:03:06.4844799Z Successfully installed torch-1.14.0a0+git1cfd385 2022-11-23T01:03:06.5482315Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2022-11-23T01:03:06.5640429Z + TORCH_INSTALL_DIR=/opt/conda/lib/python3.7/site-packages/torch 2022-11-23T01:03:06.5641314Z + TORCH_BIN_DIR=/opt/conda/lib/python3.7/site-packages/torch/bin 2022-11-23T01:03:06.5641938Z + TORCH_LIB_DIR=/opt/conda/lib/python3.7/site-packages/torch/lib 2022-11-23T01:03:06.5642582Z + TORCH_TEST_DIR=/opt/conda/lib/python3.7/site-packages/torch/test 2022-11-23T01:03:06.5642885Z + BUILD_DIR=build 2022-11-23T01:03:06.5643149Z + BUILD_RENAMED_DIR=build_renamed 2022-11-23T01:03:06.5643380Z + BUILD_BIN_DIR=build/bin 2022-11-23T01:03:06.5643621Z + export VALGRIND=ON 2022-11-23T01:03:06.5646987Z + VALGRIND=ON 2022-11-23T01:03:06.5647558Z + [[ linux-focal-py3.7-gcc7 == *clang9* ]] 2022-11-23T01:03:06.5647990Z + [[ linux-focal-py3.7-gcc7 != *bazel* ]] 2022-11-23T01:03:06.5648224Z ++ realpath build/custom_test_artifacts 2022-11-23T01:03:06.5651187Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2022-11-23T01:03:06.5653524Z ++ dirname .jenkins/pytorch/test.sh 2022-11-23T01:03:06.5658716Z + source .jenkins/pytorch/common.sh 2022-11-23T01:03:06.5661733Z +++ dirname .jenkins/pytorch/common.sh 2022-11-23T01:03:06.5667821Z ++ source .jenkins/pytorch/common_utils.sh 2022-11-23T01:03:06.5669277Z +++ declare -f -t trap_add 2022-11-23T01:03:06.5674005Z ++ set -ex 2022-11-23T01:03:06.5674517Z ++ [[ linux-focal-py3.7-gcc7 == *rocm* ]] 2022-11-23T01:03:06.5674790Z ++ BUILD_TEST_LIBTORCH=0 2022-11-23T01:03:06.5675079Z + echo 'Environment variables' 2022-11-23T01:03:06.5675269Z Environment variables 2022-11-23T01:03:06.5675446Z + env 2022-11-23T01:03:06.5681759Z INSTALLED_DB=yes 2022-11-23T01:03:06.5682323Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2022-11-23T01:03:06.5682921Z BUILD_ENVIRONMENT=linux-focal-py3.7-gcc7 2022-11-23T01:03:06.5683261Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1 2022-11-23T01:03:06.5683639Z HOSTNAME=6e200e7a94c0 2022-11-23T01:03:06.5684277Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_333d154e-1df5-438a-8d1a-5798c6f59795 2022-11-23T01:03:06.5684734Z GITHUB_ACTION=__self 2022-11-23T01:03:06.5685067Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2022-11-23T01:03:06.5685384Z GITHUB_RUN_NUMBER=67324 2022-11-23T01:03:06.5685679Z TEST_CONFIG=distributed 2022-11-23T01:03:06.5686116Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2022-11-23T01:03:06.5686763Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2022-11-23T01:03:06.5687133Z GITHUB_REF_TYPE=branch 2022-11-23T01:03:06.5687440Z TORCH_CUDA_ARCH_LIST=Maxwell 2022-11-23T01:03:06.5687765Z BASE_SHA=1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T01:03:06.5688074Z XLA_CUDA= 2022-11-23T01:03:06.5688761Z *** 2022-11-23T01:03:06.5689049Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2022-11-23T01:03:06.5689348Z GITHUB_ACTIONS=true 2022-11-23T01:03:06.5689691Z SHA1=1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T01:03:06.5690096Z GITHUB_SHA=1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T01:03:06.5690417Z UCC_HOME=/usr 2022-11-23T01:03:06.5690701Z GITHUB_REF=refs/heads/master 2022-11-23T01:03:06.5691147Z SHARD_NUMBER=2 2022-11-23T01:03:06.5691429Z GITHUB_REF_PROTECTED=true 2022-11-23T01:03:06.5691746Z HOME=/var/lib/jenkins 2022-11-23T01:03:06.5692113Z GITHUB_API_URL=https://api.github.com 2022-11-23T01:03:06.5692510Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2022-11-23T01:03:06.5692841Z INSTALLED_THRIFT= 2022-11-23T01:03:06.5693140Z UCX_COMMIT= 2022-11-23T01:03:06.5693423Z SCCACHE_S3_KEY_PREFIX=pull 2022-11-23T01:03:06.5693747Z NUM_TEST_SHARDS=2 2022-11-23T01:03:06.5694028Z UCX_HOME=/usr 2022-11-23T01:03:06.5694698Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_333d154e-1df5-438a-8d1a-5798c6f59795 2022-11-23T01:03:06.5695148Z PYTORCH_RETRY_TEST_CASES=1 2022-11-23T01:03:06.5695823Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_333d154e-1df5-438a-8d1a-5798c6f59795 2022-11-23T01:03:06.5696553Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2022-11-23T01:03:06.5696967Z GITHUB_EVENT_NAME=push 2022-11-23T01:03:06.5697268Z GITHUB_RUN_ID=3528293562 2022-11-23T01:03:06.5697983Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_333d154e-1df5-438a-8d1a-5798c6f59795 2022-11-23T01:03:06.5698480Z GITHUB_ACTOR=pytorchmergebot 2022-11-23T01:03:06.5723787Z PR_NUMBER= 2022-11-23T01:03:06.5724150Z DESIRED_CUDA= 2022-11-23T01:03:06.5724480Z GITHUB_RUN_ATTEMPT=1 2022-11-23T01:03:06.5724795Z VALGRIND=ON 2022-11-23T01:03:06.5725173Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2022-11-23T01:03:06.5725535Z TERM=xterm 2022-11-23T01:03:06.5725827Z INSTALLED_VISION=yes 2022-11-23T01:03:06.5726101Z BRANCH=master 2022-11-23T01:03:06.5726424Z OPENSSL_ROOT_DIR=/opt/openssl 2022-11-23T01:03:06.5726767Z CUDA_PATH=/usr/local/cuda 2022-11-23T01:03:06.5727425Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2022-11-23T01:03:06.5727922Z GITHUB_SERVER_URL=https://github.com 2022-11-23T01:03:06.5728253Z UCC_COMMIT= 2022-11-23T01:03:06.5728507Z INSTALLED_ANDROID= 2022-11-23T01:03:06.5728787Z SHLVL=1 2022-11-23T01:03:06.5729046Z MAX_JOBS=6 2022-11-23T01:03:06.5729317Z COMMIT_MESSAGES= 2022-11-23T01:03:06.5729622Z GITHUB_REF_NAME=master 2022-11-23T01:03:06.5730208Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2022-11-23T01:03:06.5730654Z GITHUB_JOB=test 2022-11-23T01:03:06.5731161Z GITHUB_REPOSITORY=pytorch/pytorch 2022-11-23T01:03:06.5731514Z GITHUB_RETENTION_DAYS=90 2022-11-23T01:03:06.5731819Z OPENSSL_DIR=/opt/openssl 2022-11-23T01:03:06.5732137Z GITHUB_ACTION_REPOSITORY= 2022-11-23T01:03:06.5732673Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2022-11-23T01:03:06.5733171Z GITHUB_BASE_REF= 2022-11-23T01:03:06.5733415Z CI=true 2022-11-23T01:03:06.5733728Z GITHUB_REPOSITORY_OWNER=pytorch 2022-11-23T01:03:06.5734085Z INSTALLED_PROTOBUF=yes 2022-11-23T01:03:06.5734387Z GITHUB_HEAD_REF= 2022-11-23T01:03:06.5734696Z GITHUB_ACTION_REF= 2022-11-23T01:03:06.5735240Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2022-11-23T01:03:06.5735615Z GITHUB_WORKFLOW=pull 2022-11-23T01:03:06.5735947Z DEBIAN_FRONTEND=noninteractive 2022-11-23T01:03:06.5736837Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_333d154e-1df5-438a-8d1a-5798c6f59795 2022-11-23T01:03:06.5737330Z PR_BODY= 2022-11-23T01:03:06.5737609Z _=/usr/bin/env 2022-11-23T01:03:06.5737973Z + echo 'Testing pytorch' 2022-11-23T01:03:06.5738261Z Testing pytorch 2022-11-23T01:03:06.5738587Z + export LANG=C.UTF-8 2022-11-23T01:03:06.5738906Z + LANG=C.UTF-8 2022-11-23T01:03:06.5762748Z + PR_NUMBER= 2022-11-23T01:03:06.5763142Z + [[ distributed == \d\e\f\a\u\l\t ]] 2022-11-23T01:03:06.5763548Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2022-11-23T01:03:06.5764125Z + [[ linux-focal-py3.7-gcc7 == *rocm* ]] 2022-11-23T01:03:06.5764479Z + [[ distributed == \s\l\o\w ]] 2022-11-23T01:03:06.5764994Z + [[ linux-focal-py3.7-gcc7 == *slow-gradcheck* ]] 2022-11-23T01:03:06.5765422Z + [[ linux-focal-py3.7-gcc7 == *cuda* ]] 2022-11-23T01:03:06.5765680Z + [[ linux-focal-py3.7-gcc7 == *rocm* ]] 2022-11-23T01:03:06.5765901Z + [[ distributed == *crossref* ]] 2022-11-23T01:03:06.5766106Z + [[ distributed == *dynamo* ]] 2022-11-23T01:03:06.5766303Z + [[ distributed == *inductor* ]] 2022-11-23T01:03:06.5766568Z + [[ linux-focal-py3.7-gcc7 == *rocm* ]] 2022-11-23T01:03:06.5766846Z + [[ linux-focal-py3.7-gcc7 != *-bazel-* ]] 2022-11-23T01:03:06.5767097Z + pip_install --user ninja==1.10.2 2022-11-23T01:03:06.5767386Z + pip install --progress-bar off --user ninja==1.10.2 2022-11-23T01:03:06.9678544Z Collecting ninja==1.10.2 2022-11-23T01:03:06.9863245Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2022-11-23T01:03:07.7098888Z Installing collected packages: ninja 2022-11-23T01:03:07.7185063Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2022-11-23T01:03:07.7185583Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2022-11-23T01:03:07.7249041Z Successfully installed ninja-1.10.2 2022-11-23T01:03:07.7905015Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2022-11-23T01:03:07.7906091Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2022-11-23T01:03:07.7906582Z + [[ linux-focal-py3.7-gcc7 == *asan* ]] 2022-11-23T01:03:07.7906849Z + [[ linux-focal-py3.7-gcc7 == *-tsan* ]] 2022-11-23T01:03:07.7907085Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2022-11-23T01:03:07.7907319Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2022-11-23T01:03:07.7911843Z + [[ linux-focal-py3.7-gcc7 == *tbb* ]] 2022-11-23T01:03:07.7924125Z + [[ linux-focal-py3.7-gcc7 == *libtorch* ]] 2022-11-23T01:03:07.7924465Z + [[ linux-focal-py3.7-gcc7 == *-bazel-* ]] 2022-11-23T01:03:07.7924750Z + [[ linux-focal-py3.7-gcc7 == *-tsan* ]] 2022-11-23T01:03:07.7926614Z + cd test 2022-11-23T01:03:07.7926962Z + python -c 'import torch; print(torch.__config__.show())' 2022-11-23T01:03:08.8893604Z PyTorch built with: 2022-11-23T01:03:08.8894184Z - GCC 7.5 2022-11-23T01:03:08.8894567Z - C++ Version: 201402 2022-11-23T01:03:08.8895321Z - Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications 2022-11-23T01:03:08.8896058Z - Intel(R) MKL-DNN v2.7.0 (Git Hash 650085b2f3643aad05c629425983491d63b5c289) 2022-11-23T01:03:08.8896560Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2022-11-23T01:03:08.8897075Z - LAPACK is enabled (usually provided by MKL) 2022-11-23T01:03:08.8897523Z - NNPACK is enabled 2022-11-23T01:03:08.8897927Z - CPU capability usage: AVX2 2022-11-23T01:03:08.8902605Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=1.14.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 2022-11-23T01:03:08.8905919Z 2022-11-23T01:03:09.1086636Z + cd test 2022-11-23T01:03:09.1087200Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2022-11-23T01:03:10.1931317Z ATen/Parallel: 2022-11-23T01:03:10.1947771Z at::get_num_threads() : 4 2022-11-23T01:03:10.1948211Z at::get_num_interop_threads() : 4 2022-11-23T01:03:10.1948618Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2022-11-23T01:03:10.1949002Z omp_get_max_threads() : 4 2022-11-23T01:03:10.1949825Z Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications 2022-11-23T01:03:10.1950363Z mkl_get_max_threads() : 4 2022-11-23T01:03:10.1950930Z Intel(R) MKL-DNN v2.7.0 (Git Hash 650085b2f3643aad05c629425983491d63b5c289) 2022-11-23T01:03:10.1951382Z std::thread::hardware_concurrency() : 8 2022-11-23T01:03:10.1951803Z Environment variables: 2022-11-23T01:03:10.1952288Z OMP_NUM_THREADS : [not set] 2022-11-23T01:03:10.1952647Z MKL_NUM_THREADS : [not set] 2022-11-23T01:03:10.1953028Z ATen parallel backend: OpenMP 2022-11-23T01:03:10.1953260Z 2022-11-23T01:03:10.3961933Z + [[ distributed == *backward* ]] 2022-11-23T01:03:10.3962302Z + [[ distributed == *xla* ]] 2022-11-23T01:03:10.3962522Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2022-11-23T01:03:10.3962936Z + [[ linux-focal-py3.7-gcc7 == *libtorch* ]] 2022-11-23T01:03:10.3963218Z + [[ distributed == distributed ]] 2022-11-23T01:03:10.3963403Z + install_filelock 2022-11-23T01:03:10.3963585Z + pip_install filelock 2022-11-23T01:03:10.3963894Z + pip install --progress-bar off filelock 2022-11-23T01:03:10.7753210Z Collecting filelock 2022-11-23T01:03:10.8078956Z Downloading filelock-3.8.0-py3-none-any.whl (10 kB) 2022-11-23T01:03:11.5422893Z Installing collected packages: filelock 2022-11-23T01:03:11.5706984Z Successfully installed filelock-3.8.0 2022-11-23T01:03:11.6408131Z + install_triton 2022-11-23T01:03:11.6408359Z + local commit 2022-11-23T01:03:11.6408554Z + [[ distributed == *rocm* ]] 2022-11-23T01:03:11.6409988Z ++ get_pinned_commit triton 2022-11-23T01:03:11.6410263Z ++ cat .github/ci_commit_pins/triton.txt 2022-11-23T01:03:11.6421213Z + commit=0d7e7532279e45672555e344646f5c19c3972331 2022-11-23T01:03:11.6422227Z + pip_install --user git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python 2022-11-23T01:03:11.6423065Z + pip install --progress-bar off --user git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python 2022-11-23T01:03:11.9856975Z Collecting git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python 2022-11-23T01:03:11.9863907Z Cloning https://github.com/openai/triton (to revision 0d7e7532279e45672555e344646f5c19c3972331) to /tmp/pip-req-build-v9yj1p9a 2022-11-23T01:03:11.9897713Z Running command git clone --filter=blob:none --quiet https://github.com/openai/triton /tmp/pip-req-build-v9yj1p9a 2022-11-23T01:03:12.6033952Z Running command git rev-parse -q --verify 'sha^0d7e7532279e45672555e344646f5c19c3972331' 2022-11-23T01:03:12.6067325Z Running command git fetch -q https://github.com/openai/triton 0d7e7532279e45672555e344646f5c19c3972331 2022-11-23T01:03:12.9181984Z Running command git checkout -q 0d7e7532279e45672555e344646f5c19c3972331 2022-11-23T01:03:13.0767777Z Resolved https://github.com/openai/triton to commit 0d7e7532279e45672555e344646f5c19c3972331 2022-11-23T01:03:13.0769809Z Running command git submodule update --init --recursive -q 2022-11-23T01:03:13.5675777Z Preparing metadata (setup.py) ... [?25l- done 2022-11-23T01:03:13.7277045Z [?25hCollecting cmake 2022-11-23T01:03:13.7470125Z Downloading cmake-3.25.0-py2.py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (23.7 MB) 2022-11-23T01:03:14.0915623Z Requirement already satisfied: filelock in /opt/conda/lib/python3.7/site-packages (from triton==2.0.0) (3.8.0) 2022-11-23T01:03:14.0918711Z Requirement already satisfied: torch in /opt/conda/lib/python3.7/site-packages (from triton==2.0.0) (1.14.0a0+git1cfd385) 2022-11-23T01:03:14.1136863Z Requirement already satisfied: networkx in /opt/conda/lib/python3.7/site-packages (from torch->triton==2.0.0) (2.6.3) 2022-11-23T01:03:14.1140515Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.7/site-packages (from torch->triton==2.0.0) (4.3.0) 2022-11-23T01:03:14.1143987Z Requirement already satisfied: sympy in /opt/conda/lib/python3.7/site-packages (from torch->triton==2.0.0) (1.10.1) 2022-11-23T01:03:14.1322346Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.7/site-packages (from sympy->torch->triton==2.0.0) (1.2.1) 2022-11-23T01:03:14.1386071Z Building wheels for collected packages: triton 2022-11-23T01:03:58.1449591Z Building wheel for triton (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ done 2022-11-23T01:03:58.1820332Z [?25h Created wheel for triton: filename=triton-2.0.0-cp37-cp37m-linux_x86_64.whl size=15370740 sha256=d3a136369e7498dfa2b8504a066cf3c24c58c46716c84b1aa968b3acceed001b 2022-11-23T01:03:58.1820839Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/aa/c5/3a/263a5ba27ed44796dab5f19b43135afb68beabac5d137652f7 2022-11-23T01:03:58.1844925Z Successfully built triton 2022-11-23T01:03:58.9163829Z Installing collected packages: cmake, triton 2022-11-23T01:04:00.0774121Z Successfully installed cmake-3.25.0 triton-2.0.0 2022-11-23T01:04:00.1827065Z + pip_install --user jinja2 2022-11-23T01:04:00.1827454Z + pip install --progress-bar off --user jinja2 2022-11-23T01:04:00.5656172Z Collecting jinja2 2022-11-23T01:04:00.5821464Z Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB) 2022-11-23T01:04:00.5935668Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.7/site-packages (from jinja2) (2.1.1) 2022-11-23T01:04:01.3324556Z Installing collected packages: jinja2 2022-11-23T01:04:01.4072809Z Successfully installed jinja2-3.1.2 2022-11-23T01:04:01.4812326Z + test_distributed 2022-11-23T01:04:01.4813170Z + echo 'Testing distributed python tests' 2022-11-23T01:04:01.4813467Z Testing distributed python tests 2022-11-23T01:04:01.4813859Z + python test/run_test.py --distributed-tests --shard 2 2 --verbose 2022-11-23T01:04:02.9780198Z No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda' 2022-11-23T01:04:03.1212919Z Ignoring disabled issues: [] 2022-11-23T01:04:03.1471897Z Found test time stats from artifacts 2022-11-23T01:04:03.1484112Z Selected tests: 2022-11-23T01:04:03.1484525Z distributed/algorithms/quantization/test_quantization 2022-11-23T01:04:03.1484793Z distributed/test_distributed_spawn 2022-11-23T01:04:03.1485001Z distributed/test_nccl 2022-11-23T01:04:03.1485232Z distributed/pipeline/sync/test_transparency 2022-11-23T01:04:03.1485461Z distributed/pipeline/sync/test_pipeline 2022-11-23T01:04:03.1485768Z distributed/pipeline/sync/test_phony 2022-11-23T01:04:03.1486090Z distributed/pipeline/sync/test_inplace 2022-11-23T01:04:03.1486422Z distributed/pipeline/sync/test_deferred_batch_norm 2022-11-23T01:04:03.1486864Z distributed/pipeline/sync/test_checkpoint 2022-11-23T01:04:03.1487288Z distributed/pipeline/sync/test_balance 2022-11-23T01:04:03.1487535Z distributed/pipeline/sync/skip/test_tracker 2022-11-23T01:04:03.1489699Z distributed/pipeline/sync/skip/test_portal 2022-11-23T01:04:03.1490215Z distributed/pipeline/sync/skip/test_inspect_skip_layout 2022-11-23T01:04:03.1490687Z distributed/pipeline/sync/skip/test_api 2022-11-23T01:04:03.1490948Z distributed/fsdp/test_shard_utils 2022-11-23T01:04:03.1491223Z distributed/elastic/events/lib_test 2022-11-23T01:04:03.1491600Z distributed/checkpoint/test_dedup_tensors 2022-11-23T01:04:03.1491885Z distributed/_shard/test_replicated_tensor 2022-11-23T01:04:03.1492288Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 2022-11-23T01:04:03.1492787Z distributed/_shard/sharded_tensor/ops/test_embedding 2022-11-23T01:04:03.1493197Z distributed/_shard/sharded_optim/test_sharded_optim 2022-11-23T01:04:03.1493697Z distributed/_shard/sharded_tensor/ops/test_init 2022-11-23T01:04:03.1494116Z distributed/_shard/sharded_tensor/ops/test_elementwise_ops 2022-11-23T01:04:03.1494539Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 2022-11-23T01:04:03.1495018Z distributed/_shard/sharding_plan/test_sharding_plan 2022-11-23T01:04:03.1495471Z distributed/_shard/sharded_tensor/ops/test_matrix_ops 2022-11-23T01:04:03.1495906Z distributed/test_dynamo_distributed 2022-11-23T01:04:03.1496312Z distributed/_composable/test_checkpoint 2022-11-23T01:04:03.1496673Z distributed/elastic/metrics/api_test 2022-11-23T01:04:03.1496897Z distributed/nn/jit/test_instantiator 2022-11-23T01:04:03.1497207Z distributed/checkpoint/test_utils 2022-11-23T01:04:03.1497415Z distributed/checkpoint/test_planner 2022-11-23T01:04:03.1497658Z distributed/checkpoint/test_file_system_checkpoint 2022-11-23T01:04:03.1497913Z distributed/fsdp/test_checkpoint_wrapper 2022-11-23T01:04:03.1498147Z distributed/elastic/utils/util_test 2022-11-23T01:04:03.1498351Z distributed/fsdp/test_fsdp_overlap 2022-11-23T01:04:03.1498573Z distributed/fsdp/test_fsdp_uneven 2022-11-23T01:04:03.1498790Z distributed/fsdp/test_fsdp_pure_fp16 2022-11-23T01:04:03.1499032Z distributed/_tensor/parallel/test_view_sharding_dim_change 2022-11-23T01:04:03.1499276Z distributed/fsdp/test_fsdp_fx 2022-11-23T01:04:03.1499490Z distributed/test_c10d_error_logger 2022-11-23T01:04:03.1499693Z distributed/fsdp/test_fsdp_memory 2022-11-23T01:04:03.1499918Z distributed/checkpoint/test_checkpoint 2022-11-23T01:04:03.1500156Z distributed/fsdp/test_fsdp_clip_grad_norm 2022-11-23T01:04:03.1500382Z distributed/_tensor/parallel/test_2d_parallel 2022-11-23T01:04:03.1500619Z distributed/_composable/test_replicate 2022-11-23T01:04:03.1500838Z distributed/_tensor/test_math_ops 2022-11-23T01:04:03.1501044Z distributed/_tensor/test_pointwise_ops 2022-11-23T01:04:03.1501287Z distributed/_tensor/parallel/test_parallelize_api 2022-11-23T01:04:03.1501559Z distributed/checkpoint/test_file_system_checkpoint_cpu 2022-11-23T01:04:03.1501792Z distributed/_tensor/test_api 2022-11-23T01:04:03.1501997Z distributed/_tensor/test_tp_sharding_ops 2022-11-23T01:04:03.1502224Z distributed/fsdp/test_fsdp_meta 2022-11-23T01:04:03.1502441Z distributed/_tensor/test_matrix_ops 2022-11-23T01:04:03.1502670Z distributed/fsdp/test_fsdp_sharded_grad_scaler 2022-11-23T01:04:03.1502921Z distributed/fsdp/test_fsdp_freezing_weights 2022-11-23T01:04:03.1503144Z distributed/test_c10d_spawn_gloo 2022-11-23T01:04:03.1503357Z distributed/_tensor/parallel/test_tp_style 2022-11-23T01:04:03.1503569Z distributed/test_store 2022-11-23T01:04:03.1503790Z distributed/elastic/multiprocessing/api_test 2022-11-23T01:04:03.1504009Z distributed/_tensor/test_view_ops 2022-11-23T01:04:03.1504226Z distributed/fsdp/test_fsdp_comm_hooks 2022-11-23T01:04:03.1504447Z distributed/fsdp/test_fsdp_checkpoint 2022-11-23T01:04:03.1504721Z distributed/fsdp/test_fsdp_use_orig_params 2022-11-23T01:04:03.1505036Z distributed/optim/test_zero_redundancy_optimizer 2022-11-23T01:04:03.1505315Z distributed/_tensor/test_device_mesh 2022-11-23T01:04:03.1505537Z distributed/fsdp/test_fsdp_mixed_precision 2022-11-23T01:04:03.1505776Z distributed/fsdp/test_fsdp_summon_full_params 2022-11-23T01:04:03.1506093Z distributed/fsdp/test_fsdp_optim_state 2022-11-23T01:04:03.1506315Z distributed/fsdp/test_fsdp_core 2022-11-23T01:04:03.1521820Z distributed/fsdp/test_fsdp_state_dict 2022-11-23T01:04:03.1522116Z distributed/rpc/test_faulty_agent 2022-11-23T01:04:03.1522355Z distributed/rpc/cuda/test_tensorpipe_agent 2022-11-23T01:04:03.1522574Z distributed/test_c10d_gloo 2022-11-23T01:04:03.1551804Z Prioritized test from test file changes. 2022-11-23T01:04:03.1552300Z reordering tests for PR: 2022-11-23T01:04:03.1552626Z prioritized: [] 2022-11-23T01:04:03.1562976Z the rest: ['distributed/algorithms/quantization/test_quantization', 'distributed/test_distributed_spawn', 'distributed/test_nccl', 'distributed/pipeline/sync/test_transparency', 'distributed/pipeline/sync/test_pipeline', 'distributed/pipeline/sync/test_phony', 'distributed/pipeline/sync/test_inplace', 'distributed/pipeline/sync/test_deferred_batch_norm', 'distributed/pipeline/sync/test_checkpoint', 'distributed/pipeline/sync/test_balance', 'distributed/pipeline/sync/skip/test_tracker', 'distributed/pipeline/sync/skip/test_portal', 'distributed/pipeline/sync/skip/test_inspect_skip_layout', 'distributed/pipeline/sync/skip/test_api', 'distributed/fsdp/test_shard_utils', 'distributed/elastic/events/lib_test', 'distributed/checkpoint/test_dedup_tensors', 'distributed/_shard/test_replicated_tensor', 'distributed/_shard/sharded_tensor/test_sharded_tensor_reshard', 'distributed/_shard/sharded_tensor/ops/test_embedding', 'distributed/_shard/sharded_optim/test_sharded_optim', 'distributed/_shard/sharded_tensor/ops/test_init', 'distributed/_shard/sharded_tensor/ops/test_elementwise_ops', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops', 'distributed/_shard/sharding_plan/test_sharding_plan', 'distributed/_shard/sharded_tensor/ops/test_matrix_ops', 'distributed/test_dynamo_distributed', 'distributed/_composable/test_checkpoint', 'distributed/elastic/metrics/api_test', 'distributed/nn/jit/test_instantiator', 'distributed/checkpoint/test_utils', 'distributed/checkpoint/test_planner', 'distributed/checkpoint/test_file_system_checkpoint', 'distributed/fsdp/test_checkpoint_wrapper', 'distributed/elastic/utils/util_test', 'distributed/fsdp/test_fsdp_overlap', 'distributed/fsdp/test_fsdp_uneven', 'distributed/fsdp/test_fsdp_pure_fp16', 'distributed/_tensor/parallel/test_view_sharding_dim_change', 'distributed/fsdp/test_fsdp_fx', 'distributed/test_c10d_error_logger', 'distributed/fsdp/test_fsdp_memory', 'distributed/checkpoint/test_checkpoint', 'distributed/fsdp/test_fsdp_clip_grad_norm', 'distributed/_tensor/parallel/test_2d_parallel', 'distributed/_composable/test_replicate', 'distributed/_tensor/test_math_ops', 'distributed/_tensor/test_pointwise_ops', 'distributed/_tensor/parallel/test_parallelize_api', 'distributed/checkpoint/test_file_system_checkpoint_cpu', 'distributed/_tensor/test_api', 'distributed/_tensor/test_tp_sharding_ops', 'distributed/fsdp/test_fsdp_meta', 'distributed/_tensor/test_matrix_ops', 'distributed/fsdp/test_fsdp_sharded_grad_scaler', 'distributed/fsdp/test_fsdp_freezing_weights', 'distributed/test_c10d_spawn_gloo', 'distributed/_tensor/parallel/test_tp_style', 'distributed/test_store', 'distributed/elastic/multiprocessing/api_test', 'distributed/_tensor/test_view_ops', 'distributed/fsdp/test_fsdp_comm_hooks', 'distributed/fsdp/test_fsdp_checkpoint', 'distributed/fsdp/test_fsdp_use_orig_params', 'distributed/optim/test_zero_redundancy_optimizer', 'distributed/_tensor/test_device_mesh', 'distributed/fsdp/test_fsdp_mixed_precision', 'distributed/fsdp/test_fsdp_summon_full_params', 'distributed/fsdp/test_fsdp_optim_state', 'distributed/fsdp/test_fsdp_core', 'distributed/fsdp/test_fsdp_state_dict', 'distributed/rpc/test_faulty_agent', 'distributed/rpc/cuda/test_tensorpipe_agent', 'distributed/test_c10d_gloo'] 2022-11-23T01:04:03.1569255Z 2022-11-23T01:04:03.1570044Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2022-11-23T01:04:03.2002296Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2022-11-23T01:04:03.2173956Z parallel (file granularity) tests: 2022-11-23T01:04:03.2174211Z 2022-11-23T01:04:03.2174458Z serial (file granularity) tests: 2022-11-23T01:04:03.2174869Z distributed/algorithms/quantization/test_quantization 2022-11-23T01:04:03.2175300Z distributed/test_distributed_spawn 2022-11-23T01:04:03.2178034Z distributed/test_nccl 2022-11-23T01:04:03.2178429Z distributed/pipeline/sync/test_transparency 2022-11-23T01:04:03.2178952Z distributed/pipeline/sync/test_pipeline 2022-11-23T01:04:03.2179482Z distributed/pipeline/sync/test_phony 2022-11-23T01:04:03.2179788Z distributed/pipeline/sync/test_inplace 2022-11-23T01:04:03.2180174Z distributed/pipeline/sync/test_deferred_batch_norm 2022-11-23T01:04:03.2180637Z distributed/pipeline/sync/test_checkpoint 2022-11-23T01:04:03.2180936Z distributed/pipeline/sync/test_balance 2022-11-23T01:04:03.2181168Z distributed/pipeline/sync/skip/test_tracker 2022-11-23T01:04:03.2181457Z distributed/pipeline/sync/skip/test_portal 2022-11-23T01:04:03.2182005Z distributed/pipeline/sync/skip/test_inspect_skip_layout 2022-11-23T01:04:03.2182448Z distributed/pipeline/sync/skip/test_api 2022-11-23T01:04:03.2182838Z distributed/fsdp/test_shard_utils 2022-11-23T01:04:03.2183240Z distributed/elastic/events/lib_test 2022-11-23T01:04:03.2183479Z distributed/checkpoint/test_dedup_tensors 2022-11-23T01:04:03.2183772Z distributed/_shard/test_replicated_tensor 2022-11-23T01:04:03.2184129Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 2022-11-23T01:04:03.2184573Z distributed/_shard/sharded_tensor/ops/test_embedding 2022-11-23T01:04:03.2185050Z distributed/_shard/sharded_optim/test_sharded_optim 2022-11-23T01:04:03.2185345Z distributed/_shard/sharded_tensor/ops/test_init 2022-11-23T01:04:03.2185607Z distributed/_shard/sharded_tensor/ops/test_elementwise_ops 2022-11-23T01:04:03.2185867Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 2022-11-23T01:04:03.2186126Z distributed/_shard/sharding_plan/test_sharding_plan 2022-11-23T01:04:03.2186387Z distributed/_shard/sharded_tensor/ops/test_matrix_ops 2022-11-23T01:04:03.2186610Z distributed/test_dynamo_distributed 2022-11-23T01:04:03.2186835Z distributed/_composable/test_checkpoint 2022-11-23T01:04:03.2187059Z distributed/elastic/metrics/api_test 2022-11-23T01:04:03.2187267Z distributed/nn/jit/test_instantiator 2022-11-23T01:04:03.2187484Z distributed/checkpoint/test_utils 2022-11-23T01:04:03.2187702Z distributed/checkpoint/test_planner 2022-11-23T01:04:03.2187931Z distributed/checkpoint/test_file_system_checkpoint 2022-11-23T01:04:03.2188178Z distributed/fsdp/test_checkpoint_wrapper 2022-11-23T01:04:03.2188402Z distributed/elastic/utils/util_test 2022-11-23T01:04:03.2188607Z distributed/fsdp/test_fsdp_overlap 2022-11-23T01:04:03.2188822Z distributed/fsdp/test_fsdp_uneven 2022-11-23T01:04:03.2189041Z distributed/fsdp/test_fsdp_pure_fp16 2022-11-23T01:04:03.2189280Z distributed/_tensor/parallel/test_view_sharding_dim_change 2022-11-23T01:04:03.2189518Z distributed/fsdp/test_fsdp_fx 2022-11-23T01:04:03.2189727Z distributed/test_c10d_error_logger 2022-11-23T01:04:03.2189938Z distributed/fsdp/test_fsdp_memory 2022-11-23T01:04:03.2190143Z distributed/checkpoint/test_checkpoint 2022-11-23T01:04:03.2190375Z distributed/fsdp/test_fsdp_clip_grad_norm 2022-11-23T01:04:03.2190614Z distributed/_tensor/parallel/test_2d_parallel 2022-11-23T01:04:03.2190834Z distributed/_composable/test_replicate 2022-11-23T01:04:03.2191052Z distributed/_tensor/test_math_ops 2022-11-23T01:04:03.2191271Z distributed/_tensor/test_pointwise_ops 2022-11-23T01:04:03.2191503Z distributed/_tensor/parallel/test_parallelize_api 2022-11-23T01:04:03.2191769Z distributed/checkpoint/test_file_system_checkpoint_cpu 2022-11-23T01:04:03.2192002Z distributed/_tensor/test_api 2022-11-23T01:04:03.2192289Z distributed/_tensor/test_tp_sharding_ops 2022-11-23T01:04:03.2192620Z distributed/fsdp/test_fsdp_meta 2022-11-23T01:04:03.2192838Z distributed/_tensor/test_matrix_ops 2022-11-23T01:04:03.2193062Z distributed/fsdp/test_fsdp_sharded_grad_scaler 2022-11-23T01:04:03.2193310Z distributed/fsdp/test_fsdp_freezing_weights 2022-11-23T01:04:03.2193535Z distributed/test_c10d_spawn_gloo 2022-11-23T01:04:03.2193747Z distributed/_tensor/parallel/test_tp_style 2022-11-23T01:04:03.2193959Z distributed/test_store 2022-11-23T01:04:03.2194183Z distributed/elastic/multiprocessing/api_test 2022-11-23T01:04:03.2194412Z distributed/_tensor/test_view_ops 2022-11-23T01:04:03.2194614Z distributed/fsdp/test_fsdp_comm_hooks 2022-11-23T01:04:03.2194880Z distributed/fsdp/test_fsdp_checkpoint 2022-11-23T01:04:03.2195118Z distributed/fsdp/test_fsdp_use_orig_params 2022-11-23T01:04:03.2195349Z distributed/optim/test_zero_redundancy_optimizer 2022-11-23T01:04:03.2195581Z distributed/_tensor/test_device_mesh 2022-11-23T01:04:03.2195805Z distributed/fsdp/test_fsdp_mixed_precision 2022-11-23T01:04:03.2196035Z distributed/fsdp/test_fsdp_summon_full_params 2022-11-23T01:04:03.2196266Z distributed/fsdp/test_fsdp_optim_state 2022-11-23T01:04:03.2196497Z distributed/fsdp/test_fsdp_core 2022-11-23T01:04:03.2196698Z distributed/fsdp/test_fsdp_state_dict 2022-11-23T01:04:03.2196912Z distributed/rpc/test_faulty_agent 2022-11-23T01:04:03.2197137Z distributed/rpc/cuda/test_tensorpipe_agent 2022-11-23T01:04:03.2197340Z distributed/test_c10d_gloo 2022-11-23T01:04:04.7193814Z No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda' 2022-11-23T01:04:04.7214499Z No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda' 2022-11-23T01:04:04.8439840Z Ignoring disabled issues: [] 2022-11-23T01:04:04.8601985Z Ignoring disabled issues: [] 2022-11-23T01:04:05.2193513Z Running distributed/algorithms/quantization/test_quantization ... [2022-11-23 01:04:05.218981] 2022-11-23T01:04:05.2244078Z MPI not available -- MPI backend tests will be skipped 2022-11-23T01:04:05.2244614Z Map different backends to different shards for distributed/algorithms/quantization/test_quantization: {'gloo': 1, 'nccl': 2} 2022-11-23T01:04:05.2244931Z Shard 2: test should be run in 1 2022-11-23T01:04:05.2245139Z Shard 2: gloo should be run in 1 2022-11-23T01:04:05.2247701Z Running distributed/test_distributed_spawn ... [2022-11-23 01:04:05.224612] 2022-11-23T01:04:05.2286853Z MPI not available -- MPI backend tests will be skipped 2022-11-23T01:04:05.2287394Z Map different backends to different shards for distributed/test_distributed_spawn: {'gloo': 1, 'nccl': 2, 'ucc': 1} 2022-11-23T01:04:05.2287692Z Shard 2: test should be run in 1 2022-11-23T01:04:05.2287890Z Shard 2: gloo should be run in 1 2022-11-23T01:04:05.2290922Z Running distributed/test_nccl ... [2022-11-23 01:04:05.228937] 2022-11-23T01:04:05.2294970Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_nccl.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:04:05.229341] 2022-11-23T01:04:07.4899242Z 2022-11-23T01:04:07.4899744Z Expand the folded group to see the log file of distributed/test_nccl 2022-11-23T01:04:07.4904589Z ##[group]PRINTING LOG FILE of distributed/test_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_nccl_u753vfme) 2022-11-23T01:04:07.4904954Z CUDA not available, skipping tests 2022-11-23T01:04:07.4905090Z 2022-11-23T01:04:07.4905163Z Running tests... 2022-11-23T01:04:07.4905571Z ---------------------------------------------------------------------- 2022-11-23T01:04:07.4905749Z 2022-11-23T01:04:07.4905943Z ---------------------------------------------------------------------- 2022-11-23T01:04:07.4906170Z Ran 0 tests in 0.000s 2022-11-23T01:04:07.4906293Z 2022-11-23T01:04:07.4906353Z OK 2022-11-23T01:04:07.4906444Z 2022-11-23T01:04:07.4906529Z Generating XML reports... 2022-11-23T01:04:07.4906862Z Test results will be stored in test-reports/python-unittest/distributed.test_nccl 2022-11-23T01:04:07.4907052Z 2022-11-23T01:04:07.4907287Z ##[endgroup] 2022-11-23T01:04:07.4907897Z FINISHED PRINTING LOG FILE of distributed/test_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_nccl_u753vfme) 2022-11-23T01:04:07.4908134Z 2022-11-23T01:04:07.4915609Z Running distributed/pipeline/sync/test_transparency ... [2022-11-23 01:04:07.491360] 2022-11-23T01:04:07.4918603Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_transparency.py', '-v'] ... [2022-11-23 01:04:07.491690] 2022-11-23T01:04:09.7985367Z 2022-11-23T01:04:09.7986042Z Expand the folded group to see the log file of distributed/pipeline/sync/test_transparency 2022-11-23T01:04:09.7986928Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_transparency (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_transparency_ccx90971) 2022-11-23T01:04:09.7987546Z ============================= test session starts ============================== 2022-11-23T01:04:09.7988055Z platform linux -- Python 3.7.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T01:04:09.7988358Z cachedir: .pytest_cache 2022-11-23T01:04:09.7988980Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-11-23T01:04:09.7989407Z torch: 1.14.0a0+git1cfd385 2022-11-23T01:04:09.7989650Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-11-23T01:04:09.7990087Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2 2022-11-23T01:04:09.7990386Z collecting ... collected 1 item 2022-11-23T01:04:09.7990681Z Running 1 items in this shard: test/distributed/pipeline/sync/test_transparency.py::test_simple_linears 2022-11-23T01:04:09.7990889Z 2022-11-23T01:04:09.7991052Z distributed/pipeline/sync/test_transparency.py::test_simple_linears PASSED [100%] 2022-11-23T01:04:09.7991240Z 2022-11-23T01:04:09.7991351Z ============================== 1 passed in 0.14s =============================== 2022-11-23T01:04:09.7991505Z 2022-11-23T01:04:09.7991772Z ##[endgroup] 2022-11-23T01:04:09.7992360Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_transparency (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_transparency_ccx90971) 2022-11-23T01:04:09.7992660Z 2022-11-23T01:04:09.7992878Z Running distributed/pipeline/sync/test_pipeline ... [2022-11-23 01:04:09.798768] 2022-11-23T01:04:09.7993342Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_pipeline.py', '-v'] ... [2022-11-23 01:04:09.799073] 2022-11-23T01:04:11.4931634Z 2022-11-23T01:04:11.4932248Z Expand the folded group to see the log file of distributed/pipeline/sync/test_pipeline 2022-11-23T01:04:11.4933390Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_pipeline (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_pipeline_pkra5vqw) 2022-11-23T01:04:11.4933817Z ============================= test session starts ============================== 2022-11-23T01:04:11.4934301Z platform linux -- Python 3.7.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T01:04:11.4934561Z cachedir: .pytest_cache 2022-11-23T01:04:11.4934978Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-11-23T01:04:11.4935302Z torch: 1.14.0a0+git1cfd385 2022-11-23T01:04:11.4935617Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-11-23T01:04:11.4936159Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2 2022-11-23T01:04:11.4936514Z collecting ... collected 1 item 2022-11-23T01:04:11.4937056Z Running 1 items in this shard: test/distributed/pipeline/sync/test_pipeline.py::test_clock_cycles 2022-11-23T01:04:11.4937377Z 2022-11-23T01:04:11.4937531Z distributed/pipeline/sync/test_pipeline.py::test_clock_cycles PASSED [100%] 2022-11-23T01:04:11.4937710Z 2022-11-23T01:04:11.4937806Z ============================== 1 passed in 0.02s =============================== 2022-11-23T01:04:11.4938112Z 2022-11-23T01:04:11.4938356Z ##[endgroup] 2022-11-23T01:04:11.4938892Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_pipeline (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_pipeline_pkra5vqw) 2022-11-23T01:04:11.4939288Z 2022-11-23T01:04:11.4939683Z Running distributed/pipeline/sync/test_phony ... [2022-11-23 01:04:11.493474] 2022-11-23T01:04:11.4940277Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_phony.py', '-v'] ... [2022-11-23 01:04:11.493794] 2022-11-23T01:04:13.1956593Z 2022-11-23T01:04:13.1957159Z Expand the folded group to see the log file of distributed/pipeline/sync/test_phony 2022-11-23T01:04:13.1958468Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_phony (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_phony_u5qrnw0p) 2022-11-23T01:04:13.1958880Z ============================= test session starts ============================== 2022-11-23T01:04:13.1959363Z platform linux -- Python 3.7.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T01:04:13.1959609Z cachedir: .pytest_cache 2022-11-23T01:04:13.1960288Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-11-23T01:04:13.1961044Z torch: 1.14.0a0+git1cfd385 2022-11-23T01:04:13.1961297Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-11-23T01:04:13.1961887Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2 2022-11-23T01:04:13.1962185Z collecting ... collected 4 items 2022-11-23T01:04:13.1962758Z Running 4 items in this shard: test/distributed/pipeline/sync/test_phony.py::test_phony_size, test/distributed/pipeline/sync/test_phony.py::test_phony_requires_grad, test/distributed/pipeline/sync/test_phony.py::test_cached_phony, test/distributed/pipeline/sync/test_phony.py::test_phony_in_autograd_function 2022-11-23T01:04:13.1963152Z 2022-11-23T01:04:13.1963308Z distributed/pipeline/sync/test_phony.py::test_phony_size PASSED [ 25%] 2022-11-23T01:04:13.1963618Z distributed/pipeline/sync/test_phony.py::test_phony_requires_grad PASSED [ 50%] 2022-11-23T01:04:13.1963933Z distributed/pipeline/sync/test_phony.py::test_cached_phony PASSED [ 75%] 2022-11-23T01:04:13.1964277Z distributed/pipeline/sync/test_phony.py::test_phony_in_autograd_function PASSED [100%] 2022-11-23T01:04:13.1964492Z 2022-11-23T01:04:13.1964639Z ============================== 4 passed in 0.02s =============================== 2022-11-23T01:04:13.1964802Z 2022-11-23T01:04:13.1965040Z ##[endgroup] 2022-11-23T01:04:13.1965522Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_phony (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_phony_u5qrnw0p) 2022-11-23T01:04:13.1965798Z 2022-11-23T01:04:13.1966005Z Running distributed/pipeline/sync/test_inplace ... [2022-11-23 01:04:13.196053] 2022-11-23T01:04:13.1966458Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_inplace.py', '-v'] ... [2022-11-23 01:04:13.196401] 2022-11-23T01:04:14.9601704Z 2022-11-23T01:04:14.9602338Z Expand the folded group to see the log file of distributed/pipeline/sync/test_inplace 2022-11-23T01:04:14.9603581Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_inplace (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_inplace_bpzbqs1w) 2022-11-23T01:04:14.9604334Z ============================= test session starts ============================== 2022-11-23T01:04:14.9604973Z platform linux -- Python 3.7.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T01:04:14.9605258Z cachedir: .pytest_cache 2022-11-23T01:04:14.9605752Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-11-23T01:04:14.9606096Z torch: 1.14.0a0+git1cfd385 2022-11-23T01:04:14.9606591Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-11-23T01:04:14.9607041Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2 2022-11-23T01:04:14.9607338Z collecting ... collected 3 items 2022-11-23T01:04:14.9607801Z Running 3 items in this shard: test/distributed/pipeline/sync/test_inplace.py::test_inplace_on_requires_grad, test/distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad, test/distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad 2022-11-23T01:04:14.9608159Z 2022-11-23T01:04:14.9608323Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_requires_grad PASSED [ 33%] 2022-11-23T01:04:14.9608791Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad XFAIL [ 66%] 2022-11-23T01:04:14.9609270Z distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad XFAIL [100%] 2022-11-23T01:04:14.9609511Z 2022-11-23T01:04:14.9609622Z ========================= 1 passed, 2 xfailed in 0.09s ========================= 2022-11-23T01:04:14.9609765Z 2022-11-23T01:04:14.9610016Z ##[endgroup] 2022-11-23T01:04:14.9610519Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_inplace (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_inplace_bpzbqs1w) 2022-11-23T01:04:14.9610803Z 2022-11-23T01:04:14.9611042Z Running distributed/pipeline/sync/test_deferred_batch_norm ... [2022-11-23 01:04:14.960483] 2022-11-23T01:04:14.9611524Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_deferred_batch_norm.py', '-v'] ... [2022-11-23 01:04:14.960810] 2022-11-23T01:04:17.3249707Z 2022-11-23T01:04:17.3250257Z Expand the folded group to see the log file of distributed/pipeline/sync/test_deferred_batch_norm 2022-11-23T01:04:17.3251722Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_deferred_batch_norm (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_deferred_batch_norm_e91y183y) 2022-11-23T01:04:17.3252529Z ============================= test session starts ============================== 2022-11-23T01:04:17.3253288Z platform linux -- Python 3.7.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T01:04:17.3253550Z cachedir: .pytest_cache 2022-11-23T01:04:17.3254049Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-11-23T01:04:17.3254395Z torch: 1.14.0a0+git1cfd385 2022-11-23T01:04:17.3254679Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-11-23T01:04:17.3255096Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2 2022-11-23T01:04:17.3255400Z collecting ... collected 11 items 2022-11-23T01:04:17.3257161Z Running 11 items in this shard: test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-1], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-4], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-1], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-4], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[0.1], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[None], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_convert_deferred_batch_norm, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_eval, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_optimize, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_conv_bn, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_input_requiring_grad 2022-11-23T01:04:17.3258209Z 2022-11-23T01:04:17.3258465Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-1] PASSED [ 9%] 2022-11-23T01:04:17.3258899Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-4] PASSED [ 18%] 2022-11-23T01:04:17.3259507Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-1] PASSED [ 27%] 2022-11-23T01:04:17.3259931Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-4] PASSED [ 36%] 2022-11-23T01:04:17.3260286Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[0.1] PASSED [ 45%] 2022-11-23T01:04:17.3260636Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[None] PASSED [ 54%] 2022-11-23T01:04:17.3260997Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_convert_deferred_batch_norm PASSED [ 63%] 2022-11-23T01:04:17.3261332Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_eval PASSED [ 72%] 2022-11-23T01:04:17.3261726Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_optimize PASSED [ 81%] 2022-11-23T01:04:17.3262054Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_conv_bn PASSED [ 90%] 2022-11-23T01:04:17.3262381Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_input_requiring_grad PASSED [100%] 2022-11-23T01:04:17.3262578Z 2022-11-23T01:04:17.3262690Z ============================== 11 passed in 0.69s ============================== 2022-11-23T01:04:17.3262826Z 2022-11-23T01:04:17.3263073Z ##[endgroup] 2022-11-23T01:04:17.3263600Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_deferred_batch_norm (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_deferred_batch_norm_e91y183y) 2022-11-23T01:04:17.3263892Z 2022-11-23T01:04:17.3264311Z Running distributed/pipeline/sync/test_checkpoint ... [2022-11-23 01:04:17.325268] 2022-11-23T01:04:17.3264794Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_checkpoint.py', '-v'] ... [2022-11-23 01:04:17.325608] 2022-11-23T01:04:19.0428836Z 2022-11-23T01:04:19.0429485Z Expand the folded group to see the log file of distributed/pipeline/sync/test_checkpoint 2022-11-23T01:04:19.0430767Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_checkpoint_55iwf50p) 2022-11-23T01:04:19.0431355Z ============================= test session starts ============================== 2022-11-23T01:04:19.0431825Z platform linux -- Python 3.7.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T01:04:19.0432131Z cachedir: .pytest_cache 2022-11-23T01:04:19.0432571Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-11-23T01:04:19.0432911Z torch: 1.14.0a0+git1cfd385 2022-11-23T01:04:19.0433261Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-11-23T01:04:19.0433772Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2 2022-11-23T01:04:19.0434072Z collecting ... collected 7 items 2022-11-23T01:04:19.0434933Z Running 7 items in this shard: test/distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cpu], test/distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad, test/distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad_with_parameter, test/distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cpu], test/distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing, test/distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing_without_checkpoint, test/distributed/pipeline/sync/test_checkpoint.py::test_non_grad_output 2022-11-23T01:04:19.0435659Z 2022-11-23T01:04:19.0435827Z distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cpu] PASSED [ 14%] 2022-11-23T01:04:19.0436170Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad PASSED [ 28%] 2022-11-23T01:04:19.0436601Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad_with_parameter PASSED [ 42%] 2022-11-23T01:04:19.0436960Z distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cpu] PASSED [ 57%] 2022-11-23T01:04:19.0437498Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing PASSED [ 71%] 2022-11-23T01:04:19.0437887Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing_without_checkpoint PASSED [ 85%] 2022-11-23T01:04:19.0438259Z distributed/pipeline/sync/test_checkpoint.py::test_non_grad_output PASSED [100%] 2022-11-23T01:04:19.0438442Z 2022-11-23T01:04:19.0438550Z ============================== 7 passed in 0.03s =============================== 2022-11-23T01:04:19.0438688Z 2022-11-23T01:04:19.0438935Z ##[endgroup] 2022-11-23T01:04:19.0439427Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_checkpoint_55iwf50p) 2022-11-23T01:04:19.0439775Z 2022-11-23T01:04:19.0439997Z Running distributed/pipeline/sync/test_balance ... [2022-11-23 01:04:19.043211] 2022-11-23T01:04:19.0440458Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_balance.py', '-v'] ... [2022-11-23 01:04:19.043541] 2022-11-23T01:04:24.7628427Z 2022-11-23T01:04:24.7629398Z Expand the folded group to see the log file of distributed/pipeline/sync/test_balance 2022-11-23T01:04:24.7630662Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_balance (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_balance_fuzlhr9e) 2022-11-23T01:04:24.7631373Z ============================= test session starts ============================== 2022-11-23T01:04:24.7632111Z platform linux -- Python 3.7.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T01:04:24.7632498Z cachedir: .pytest_cache 2022-11-23T01:04:24.7633243Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-11-23T01:04:24.7633764Z torch: 1.14.0a0+git1cfd385 2022-11-23T01:04:24.7634103Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-11-23T01:04:24.7634830Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2 2022-11-23T01:04:24.7635322Z collecting ... collected 15 items 2022-11-23T01:04:24.7637596Z Running 15 items in this shard: test/distributed/pipeline/sync/test_balance.py::test_blockpartition, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_zeros, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_non_positive_partitions, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_short_sequence, test/distributed/pipeline/sync/test_balance.py::test_balance_by_time[cpu], test/distributed/pipeline/sync/test_balance.py::test_balance_by_time_loop_resets_input, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_latent, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_param, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_param_scale, test/distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cpu], test/distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cpu], test/distributed/pipeline/sync/test_balance.py::test_not_training, test/distributed/pipeline/sync/test_balance.py::test_balance_by_time_tuple, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_tuple, test/distributed/pipeline/sync/test_balance.py::test_already_has_grad 2022-11-23T01:04:24.7639528Z 2022-11-23T01:04:24.7639748Z distributed/pipeline/sync/test_balance.py::test_blockpartition PASSED [ 6%] 2022-11-23T01:04:24.7640221Z distributed/pipeline/sync/test_balance.py::test_blockpartition_zeros PASSED [ 13%] 2022-11-23T01:04:24.7640983Z distributed/pipeline/sync/test_balance.py::test_blockpartition_non_positive_partitions PASSED [ 20%] 2022-11-23T01:04:24.7641547Z distributed/pipeline/sync/test_balance.py::test_blockpartition_short_sequence PASSED [ 26%] 2022-11-23T01:04:24.7642068Z distributed/pipeline/sync/test_balance.py::test_balance_by_time[cpu] SKIPPED [ 33%] 2022-11-23T01:04:24.7642832Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_loop_resets_input PASSED [ 40%] 2022-11-23T01:04:24.7643357Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_latent SKIPPED [ 46%] 2022-11-23T01:04:24.7643886Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param SKIPPED [ 53%] 2022-11-23T01:04:24.7644373Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param_scale SKIPPED [ 60%] 2022-11-23T01:04:24.7644844Z distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cpu] PASSED [ 66%] 2022-11-23T01:04:24.7645324Z distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cpu] PASSED [ 73%] 2022-11-23T01:04:24.7645870Z distributed/pipeline/sync/test_balance.py::test_not_training PASSED [ 80%] 2022-11-23T01:04:24.7646319Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_tuple PASSED [ 86%] 2022-11-23T01:04:24.7646789Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_tuple SKIPPED [ 93%] 2022-11-23T01:04:24.7647242Z distributed/pipeline/sync/test_balance.py::test_already_has_grad PASSED [100%] 2022-11-23T01:04:24.7647469Z 2022-11-23T01:04:24.7647626Z ======================== 10 passed, 5 skipped in 4.05s ========================= 2022-11-23T01:04:24.7647818Z 2022-11-23T01:04:24.7648193Z ##[endgroup] 2022-11-23T01:04:24.7648936Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_balance (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_balance_fuzlhr9e) 2022-11-23T01:04:24.7649325Z 2022-11-23T01:04:24.7649655Z Running distributed/pipeline/sync/skip/test_tracker ... [2022-11-23 01:04:24.763180] 2022-11-23T01:04:24.7650358Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_tracker.py', '-v'] ... [2022-11-23 01:04:24.763582] 2022-11-23T01:04:26.4734088Z 2022-11-23T01:04:26.4734754Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_tracker 2022-11-23T01:04:26.4735993Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_tracker (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_tracker_zkqywhmv) 2022-11-23T01:04:26.4736433Z ============================= test session starts ============================== 2022-11-23T01:04:26.4736907Z platform linux -- Python 3.7.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T01:04:26.4737164Z cachedir: .pytest_cache 2022-11-23T01:04:26.4737586Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-11-23T01:04:26.4737911Z torch: 1.14.0a0+git1cfd385 2022-11-23T01:04:26.4738189Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-11-23T01:04:26.4738611Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2 2022-11-23T01:04:26.4738909Z collecting ... collected 6 items 2022-11-23T01:04:26.4739651Z Running 6 items in this shard: test/distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker, test/distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker_by_data_parallel, test/distributed/pipeline/sync/skip/test_tracker.py::test_reuse_portal, test/distributed/pipeline/sync/skip/test_tracker.py::test_no_copy_no_portal, test/distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_without_checkpointing, test/distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_with_checkpointing 2022-11-23T01:04:26.4740259Z 2022-11-23T01:04:26.4740425Z distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker PASSED [ 16%] 2022-11-23T01:04:26.4740796Z distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker_by_data_parallel SKIPPED [ 33%] 2022-11-23T01:04:26.4741174Z distributed/pipeline/sync/skip/test_tracker.py::test_reuse_portal PASSED [ 50%] 2022-11-23T01:04:26.4741554Z distributed/pipeline/sync/skip/test_tracker.py::test_no_copy_no_portal PASSED [ 66%] 2022-11-23T01:04:26.4742127Z distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_without_checkpointing PASSED [ 83%] 2022-11-23T01:04:26.4742501Z distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_with_checkpointing PASSED [100%] 2022-11-23T01:04:26.4742689Z 2022-11-23T01:04:26.4742808Z ========================= 5 passed, 1 skipped in 0.03s ========================= 2022-11-23T01:04:26.4742952Z 2022-11-23T01:04:26.4744492Z ##[endgroup] 2022-11-23T01:04:26.4745147Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_tracker (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_tracker_zkqywhmv) 2022-11-23T01:04:26.4745457Z 2022-11-23T01:04:26.4745680Z Running distributed/pipeline/sync/skip/test_portal ... [2022-11-23 01:04:26.473766] 2022-11-23T01:04:26.4746293Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_portal.py', '-v'] ... [2022-11-23 01:04:26.474088] 2022-11-23T01:04:28.2048818Z 2022-11-23T01:04:28.2049303Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_portal 2022-11-23T01:04:28.2050551Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_portal (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_portal_as61cscf) 2022-11-23T01:04:28.2051305Z ============================= test session starts ============================== 2022-11-23T01:04:28.2052063Z platform linux -- Python 3.7.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T01:04:28.2052488Z cachedir: .pytest_cache 2022-11-23T01:04:28.2053182Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-11-23T01:04:28.2053685Z torch: 1.14.0a0+git1cfd385 2022-11-23T01:04:28.2054079Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-11-23T01:04:28.2054771Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2 2022-11-23T01:04:28.2055215Z collecting ... collected 10 items 2022-11-23T01:04:28.2056982Z Running 10 items in this shard: test/distributed/pipeline/sync/skip/test_portal.py::test_copy_returns_on_next_device, test/distributed/pipeline/sync/skip/test_portal.py::test_blue_orange, test/distributed/pipeline/sync/skip/test_portal.py::test_blue_orange_not_requires_grad, test/distributed/pipeline/sync/skip/test_portal.py::test_use_grad, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_0, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_1, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_2, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_4, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3_plus_1 2022-11-23T01:04:28.2058476Z 2022-11-23T01:04:28.2058769Z distributed/pipeline/sync/skip/test_portal.py::test_copy_returns_on_next_device SKIPPED [ 10%] 2022-11-23T01:04:28.2059282Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange PASSED [ 20%] 2022-11-23T01:04:28.2059743Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange_not_requires_grad PASSED [ 30%] 2022-11-23T01:04:28.2060190Z distributed/pipeline/sync/skip/test_portal.py::test_use_grad PASSED [ 40%] 2022-11-23T01:04:28.2060654Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_0 PASSED [ 50%] 2022-11-23T01:04:28.2061164Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_1 PASSED [ 60%] 2022-11-23T01:04:28.2061650Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_2 PASSED [ 70%] 2022-11-23T01:04:28.2062176Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3 PASSED [ 80%] 2022-11-23T01:04:28.2062710Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_4 PASSED [ 90%] 2022-11-23T01:04:28.2063460Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3_plus_1 PASSED [100%] 2022-11-23T01:04:28.2063751Z 2022-11-23T01:04:28.2063926Z ========================= 9 passed, 1 skipped in 0.04s ========================= 2022-11-23T01:04:28.2064130Z 2022-11-23T01:04:28.2064508Z ##[endgroup] 2022-11-23T01:04:28.2065305Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_portal (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_portal_as61cscf) 2022-11-23T01:04:28.2065714Z 2022-11-23T01:04:28.2066052Z Running distributed/pipeline/sync/skip/test_inspect_skip_layout ... [2022-11-23 01:04:28.205305] 2022-11-23T01:04:28.2066862Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_inspect_skip_layout.py', '-v'] ... [2022-11-23 01:04:28.205695] 2022-11-23T01:04:29.9045726Z 2022-11-23T01:04:29.9046409Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_inspect_skip_layout 2022-11-23T01:04:29.9047425Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_inspect_skip_layout (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_inspect_skip_layout_h713bb9_) 2022-11-23T01:04:29.9047892Z ============================= test session starts ============================== 2022-11-23T01:04:29.9048345Z platform linux -- Python 3.7.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T01:04:29.9048612Z cachedir: .pytest_cache 2022-11-23T01:04:29.9049044Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-11-23T01:04:29.9049375Z torch: 1.14.0a0+git1cfd385 2022-11-23T01:04:29.9049599Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-11-23T01:04:29.9050111Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2 2022-11-23T01:04:29.9050408Z collecting ... collected 6 items 2022-11-23T01:04:29.9051166Z Running 6 items in this shard: test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_no_skippables, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_inner_partition, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_adjoining_partitions, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_far_partitions, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_pop_2_from_different_partitions, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_namespace 2022-11-23T01:04:29.9051807Z 2022-11-23T01:04:29.9051967Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_no_skippables PASSED [ 16%] 2022-11-23T01:04:29.9052330Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_inner_partition PASSED [ 33%] 2022-11-23T01:04:29.9052694Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_adjoining_partitions PASSED [ 50%] 2022-11-23T01:04:29.9053135Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_far_partitions PASSED [ 66%] 2022-11-23T01:04:29.9053500Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_pop_2_from_different_partitions PASSED [ 83%] 2022-11-23T01:04:29.9053868Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_namespace PASSED [100%] 2022-11-23T01:04:29.9054056Z 2022-11-23T01:04:29.9054166Z ============================== 6 passed in 0.03s =============================== 2022-11-23T01:04:29.9054301Z 2022-11-23T01:04:29.9054505Z ##[endgroup] 2022-11-23T01:04:29.9055045Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_inspect_skip_layout (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_inspect_skip_layout_h713bb9_) 2022-11-23T01:04:29.9055370Z 2022-11-23T01:04:29.9055582Z Running distributed/pipeline/sync/skip/test_api ... [2022-11-23 01:04:29.904909] 2022-11-23T01:04:29.9056219Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_api.py', '-v'] ... [2022-11-23 01:04:29.905227] 2022-11-23T01:04:31.6088603Z 2022-11-23T01:04:31.6089272Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_api 2022-11-23T01:04:31.6090259Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_api (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_api_heb2dg1t) 2022-11-23T01:04:31.6090691Z ============================= test session starts ============================== 2022-11-23T01:04:31.6091167Z platform linux -- Python 3.7.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T01:04:31.6091426Z cachedir: .pytest_cache 2022-11-23T01:04:31.6092173Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-11-23T01:04:31.6092500Z torch: 1.14.0a0+git1cfd385 2022-11-23T01:04:31.6092805Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-11-23T01:04:31.6093285Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2 2022-11-23T01:04:31.6093581Z collecting ... collected 3 items 2022-11-23T01:04:31.6094037Z Running 3 items in this shard: test/distributed/pipeline/sync/skip/test_api.py::test_namespace_difference, test/distributed/pipeline/sync/skip/test_api.py::test_namespace_copy, test/distributed/pipeline/sync/skip/test_api.py::test_skippable_repr 2022-11-23T01:04:31.6094386Z 2022-11-23T01:04:31.6094548Z distributed/pipeline/sync/skip/test_api.py::test_namespace_difference PASSED [ 33%] 2022-11-23T01:04:31.6094881Z distributed/pipeline/sync/skip/test_api.py::test_namespace_copy PASSED [ 66%] 2022-11-23T01:04:31.6095301Z distributed/pipeline/sync/skip/test_api.py::test_skippable_repr PASSED [100%] 2022-11-23T01:04:31.6095480Z 2022-11-23T01:04:31.6095590Z ============================== 3 passed in 0.02s =============================== 2022-11-23T01:04:31.6095726Z 2022-11-23T01:04:31.6096000Z ##[endgroup] 2022-11-23T01:04:31.6096515Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_api (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_api_heb2dg1t) 2022-11-23T01:04:31.6096805Z 2022-11-23T01:04:31.6097009Z Running distributed/fsdp/test_shard_utils ... [2022-11-23 01:04:31.609135] 2022-11-23T01:04:31.6097523Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_shard_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:04:31.609457] 2022-11-23T01:04:33.0861126Z 2022-11-23T01:04:33.0861608Z Expand the folded group to see the log file of distributed/fsdp/test_shard_utils 2022-11-23T01:04:33.0862360Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_shard_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_shard_utils_ckd39uht) 2022-11-23T01:04:33.0862631Z 2022-11-23T01:04:33.0862848Z ##[endgroup] 2022-11-23T01:04:33.0863418Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_shard_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_shard_utils_ckd39uht) 2022-11-23T01:04:33.0863677Z 2022-11-23T01:04:33.0865320Z Running distributed/elastic/events/lib_test ... [2022-11-23 01:04:33.086336] 2022-11-23T01:04:33.0868379Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/elastic/events/lib_test.py', '-v'] ... [2022-11-23 01:04:33.086661] 2022-11-23T01:04:34.8631643Z 2022-11-23T01:04:34.8632199Z Expand the folded group to see the log file of distributed/elastic/events/lib_test 2022-11-23T01:04:34.8633014Z ##[group]PRINTING LOG FILE of distributed/elastic/events/lib_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-events-lib_test_kuj1ql0v) 2022-11-23T01:04:34.8633421Z ============================= test session starts ============================== 2022-11-23T01:04:34.8633915Z platform linux -- Python 3.7.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T01:04:34.8634160Z cachedir: .pytest_cache 2022-11-23T01:04:34.8634825Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-11-23T01:04:34.8635259Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-11-23T01:04:34.8635779Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2 2022-11-23T01:04:34.8636122Z collecting ... collected 8 items 2022-11-23T01:04:34.8637155Z Running 8 items in this shard: test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_created, test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_deser, test/distributed/elastic/events/lib_test.py::EventLibTest::test_get_or_create_logger, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event_does_not_run_if_invalid_dest, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_created, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_deserialize, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_str 2022-11-23T01:04:34.8638127Z 2022-11-23T01:04:34.8638346Z distributed/elastic/events/lib_test.py::EventLibTest::test_event_created PASSED [ 12%] 2022-11-23T01:04:34.8638689Z distributed/elastic/events/lib_test.py::EventLibTest::test_event_deser PASSED [ 25%] 2022-11-23T01:04:34.8639086Z distributed/elastic/events/lib_test.py::EventLibTest::test_get_or_create_logger PASSED [ 37%] 2022-11-23T01:04:34.8639449Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event PASSED [ 50%] 2022-11-23T01:04:34.8639926Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event_does_not_run_if_invalid_dest PASSED [ 62%] 2022-11-23T01:04:34.8640323Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_created PASSED [ 75%] 2022-11-23T01:04:34.8640922Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_deserialize PASSED [ 87%] 2022-11-23T01:04:34.8641318Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_str PASSED [100%] 2022-11-23T01:04:34.8641516Z 2022-11-23T01:04:34.8641628Z ============================== 8 passed in 1.18s =============================== 2022-11-23T01:04:34.8641765Z 2022-11-23T01:04:34.8642013Z ##[endgroup] 2022-11-23T01:04:34.8642515Z FINISHED PRINTING LOG FILE of distributed/elastic/events/lib_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-events-lib_test_kuj1ql0v) 2022-11-23T01:04:34.8642827Z 2022-11-23T01:04:34.8643064Z Running distributed/checkpoint/test_dedup_tensors ... [2022-11-23 01:04:34.863443] 2022-11-23T01:04:34.8643648Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_dedup_tensors.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:04:34.863769] 2022-11-23T01:04:36.6752878Z 2022-11-23T01:04:36.6753544Z Expand the folded group to see the log file of distributed/checkpoint/test_dedup_tensors 2022-11-23T01:04:36.6754769Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_dedup_tensors (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_dedup_tensors_f0g06nac) 2022-11-23T01:04:36.6755495Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_dedup_tensors 2022-11-23T01:04:36.6755723Z 2022-11-23T01:04:36.6755798Z Running tests... 2022-11-23T01:04:36.6756109Z ---------------------------------------------------------------------- 2022-11-23T01:04:36.6756407Z test_dedup_shards (__main__.TestDedupTensor) ... ok (0.263s) 2022-11-23T01:04:36.6756558Z 2022-11-23T01:04:36.6756757Z ---------------------------------------------------------------------- 2022-11-23T01:04:36.6756996Z Ran 1 test in 0.263s 2022-11-23T01:04:36.6757163Z 2022-11-23T01:04:36.6757229Z OK 2022-11-23T01:04:36.6757321Z 2022-11-23T01:04:36.6757392Z Generating XML reports... 2022-11-23T01:04:36.6758197Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_dedup_tensors/TEST-TestDedupTensor-20221123010436.xml 2022-11-23T01:04:36.6758473Z 2022-11-23T01:04:36.6758721Z ##[endgroup] 2022-11-23T01:04:36.6759193Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_dedup_tensors (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_dedup_tensors_f0g06nac) 2022-11-23T01:04:36.6759476Z 2022-11-23T01:04:36.6759698Z Running distributed/_shard/test_replicated_tensor ... [2022-11-23 01:04:36.675623] 2022-11-23T01:04:36.6761793Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/test_replicated_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:04:36.675990] 2022-11-23T01:04:38.1346470Z 2022-11-23T01:04:38.1347013Z Expand the folded group to see the log file of distributed/_shard/test_replicated_tensor 2022-11-23T01:04:38.1347867Z ##[group]PRINTING LOG FILE of distributed/_shard/test_replicated_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_replicated_tensor_wmzyhtru) 2022-11-23T01:04:38.1348154Z 2022-11-23T01:04:38.1348370Z ##[endgroup] 2022-11-23T01:04:38.1348926Z FINISHED PRINTING LOG FILE of distributed/_shard/test_replicated_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_replicated_tensor_wmzyhtru) 2022-11-23T01:04:38.1349198Z 2022-11-23T01:04:38.1350203Z Running distributed/_shard/sharded_tensor/test_sharded_tensor_reshard ... [2022-11-23 01:04:38.134851] 2022-11-23T01:04:38.1353992Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_sharded_tensor_reshard.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:04:38.135190] 2022-11-23T01:04:39.6188763Z 2022-11-23T01:04:39.6189419Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 2022-11-23T01:04:39.6190916Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_sharded_tensor_reshard (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_sharded_tensor_reshard_h5kfnmue) 2022-11-23T01:04:39.6191626Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard 2022-11-23T01:04:39.6191856Z 2022-11-23T01:04:39.6191933Z Running tests... 2022-11-23T01:04:39.6192310Z ---------------------------------------------------------------------- 2022-11-23T01:04:39.6192667Z test_sharded_tensor_reshard (__main__.TestReshard) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:39.6193187Z test_sharded_tensor_reshard_errors (__main__.TestReshard) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:39.6193403Z 2022-11-23T01:04:39.6193603Z ---------------------------------------------------------------------- 2022-11-23T01:04:39.6193942Z Ran 2 tests in 0.002s 2022-11-23T01:04:39.6194092Z 2022-11-23T01:04:39.6194167Z OK (skipped=2) 2022-11-23T01:04:39.6194265Z 2022-11-23T01:04:39.6194351Z Generating XML reports... 2022-11-23T01:04:39.6194844Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard/TEST-TestReshard-20221123010439.xml 2022-11-23T01:04:39.6195112Z 2022-11-23T01:04:39.6195353Z ##[endgroup] 2022-11-23T01:04:39.6195879Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_sharded_tensor_reshard (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_sharded_tensor_reshard_h5kfnmue) 2022-11-23T01:04:39.6196194Z 2022-11-23T01:04:39.6196494Z Running distributed/_shard/sharded_tensor/ops/test_embedding ... [2022-11-23 01:04:39.619229] 2022-11-23T01:04:39.6197628Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_embedding.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:04:39.619592] 2022-11-23T01:04:41.0897005Z 2022-11-23T01:04:41.0897740Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_embedding 2022-11-23T01:04:41.0899123Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_embedding (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_embedding_vm3w68t0) 2022-11-23T01:04:41.0899817Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding 2022-11-23T01:04:41.0900041Z 2022-11-23T01:04:41.0900117Z Running tests... 2022-11-23T01:04:41.0900423Z ---------------------------------------------------------------------- 2022-11-23T01:04:41.0900799Z test_sharded_embedding_colwise (__main__.TestShardedEmbedding) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:41.0901285Z test_sharded_embedding_rowwise (__main__.TestShardedEmbedding) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-11-23T01:04:41.0901498Z 2022-11-23T01:04:41.0901700Z ---------------------------------------------------------------------- 2022-11-23T01:04:41.0901943Z Ran 2 tests in 0.004s 2022-11-23T01:04:41.0902059Z 2022-11-23T01:04:41.0902119Z OK (skipped=2) 2022-11-23T01:04:41.0902229Z 2022-11-23T01:04:41.0902314Z Generating XML reports... 2022-11-23T01:04:41.0902961Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding/TEST-TestShardedEmbedding-20221123010440.xml 2022-11-23T01:04:41.0903248Z 2022-11-23T01:04:41.0903467Z ##[endgroup] 2022-11-23T01:04:41.0903988Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_embedding (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_embedding_vm3w68t0) 2022-11-23T01:04:41.0904289Z 2022-11-23T01:04:41.0904708Z Running distributed/_shard/sharded_optim/test_sharded_optim ... [2022-11-23 01:04:41.090119] 2022-11-23T01:04:41.0906650Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_optim/test_sharded_optim.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:04:41.090473] 2022-11-23T01:04:42.5571646Z 2022-11-23T01:04:42.5572729Z Expand the folded group to see the log file of distributed/_shard/sharded_optim/test_sharded_optim 2022-11-23T01:04:42.5573680Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_optim/test_sharded_optim (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_optim-test_sharded_optim_qeayztzr) 2022-11-23T01:04:42.5574378Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim 2022-11-23T01:04:42.5574606Z 2022-11-23T01:04:42.5574668Z Running tests... 2022-11-23T01:04:42.5575020Z ---------------------------------------------------------------------- 2022-11-23T01:04:42.5575506Z test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:42.5576121Z test_sharded_optim (__main__.TestShardedOptimizer) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:42.5576318Z 2022-11-23T01:04:42.5576522Z ---------------------------------------------------------------------- 2022-11-23T01:04:42.5576766Z Ran 2 tests in 0.003s 2022-11-23T01:04:42.5576881Z 2022-11-23T01:04:42.5576958Z OK (skipped=2) 2022-11-23T01:04:42.5577066Z 2022-11-23T01:04:42.5577148Z Generating XML reports... 2022-11-23T01:04:42.5577710Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim/TEST-TestShardedOptimizer-20221123010442.xml 2022-11-23T01:04:42.5577996Z 2022-11-23T01:04:42.5578237Z ##[endgroup] 2022-11-23T01:04:42.5578732Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_optim/test_sharded_optim (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_optim-test_sharded_optim_qeayztzr) 2022-11-23T01:04:42.5579032Z 2022-11-23T01:04:42.5579288Z Running distributed/_shard/sharded_tensor/ops/test_init ... [2022-11-23 01:04:42.557460] 2022-11-23T01:04:42.5581058Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_init.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:04:42.557888] 2022-11-23T01:04:44.0286097Z 2022-11-23T01:04:44.0286634Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_init 2022-11-23T01:04:44.0287883Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_init (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_init_heamhilz) 2022-11-23T01:04:44.0288795Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init 2022-11-23T01:04:44.0289439Z 2022-11-23T01:04:44.0289572Z Running tests... 2022-11-23T01:04:44.0290128Z ---------------------------------------------------------------------- 2022-11-23T01:04:44.0290535Z test_init_sharded_tensor_with_kaiming_uniform (__main__.TestShardedTensorNNInit) 2022-11-23T01:04:44.0290994Z Test torch.nn.init.kaiming_uniform_(ShardedTensor, a, mode, nonlinearit) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:44.0291462Z test_init_sharded_tensor_with_normal (__main__.TestShardedTensorNNInit) 2022-11-23T01:04:44.0291815Z Test torch.nn.init.normal_(ShardedTensor, mean, std) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:44.0292169Z test_init_sharded_tensor_with_uniform (__main__.TestShardedTensorNNInit) 2022-11-23T01:04:44.0292523Z Test torch.nn.init.uniform_(ShardedTensor, a, b) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:44.0292730Z 2022-11-23T01:04:44.0292922Z ---------------------------------------------------------------------- 2022-11-23T01:04:44.0293166Z Ran 3 tests in 0.003s 2022-11-23T01:04:44.0293281Z 2022-11-23T01:04:44.0293355Z OK (skipped=3) 2022-11-23T01:04:44.0293465Z 2022-11-23T01:04:44.0293553Z Generating XML reports... 2022-11-23T01:04:44.0294045Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init/TEST-TestShardedTensorNNInit-20221123010443.xml 2022-11-23T01:04:44.0294372Z 2022-11-23T01:04:44.0294616Z ##[endgroup] 2022-11-23T01:04:44.0295164Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_init (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_init_heamhilz) 2022-11-23T01:04:44.0295451Z 2022-11-23T01:04:44.0295685Z Running distributed/_shard/sharded_tensor/ops/test_elementwise_ops ... [2022-11-23 01:04:44.028950] 2022-11-23T01:04:44.0296269Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_elementwise_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:04:44.029355] 2022-11-23T01:04:45.4943864Z 2022-11-23T01:04:45.4944503Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_elementwise_ops 2022-11-23T01:04:45.4946006Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_elementwise_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_elementwise_ops_rfx2tptg) 2022-11-23T01:04:45.4946736Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops 2022-11-23T01:04:45.4946965Z 2022-11-23T01:04:45.4947043Z Running tests... 2022-11-23T01:04:45.4947400Z ---------------------------------------------------------------------- 2022-11-23T01:04:45.4947789Z test_sharded_dropout (__main__.TestShardedTensorElementWiseOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:45.4948280Z test_sharded_gelu (__main__.TestShardedTensorElementWiseOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:45.4948764Z test_sharded_relu (__main__.TestShardedTensorElementWiseOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:45.4949546Z test_sharded_tensor_nan_to_num (__main__.TestShardedTensorElementWiseOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:45.4949991Z 2022-11-23T01:04:45.4950237Z ---------------------------------------------------------------------- 2022-11-23T01:04:45.4950473Z Ran 4 tests in 0.003s 2022-11-23T01:04:45.4950589Z 2022-11-23T01:04:45.4950666Z OK (skipped=4) 2022-11-23T01:04:45.4950774Z 2022-11-23T01:04:45.4950860Z Generating XML reports... 2022-11-23T01:04:45.4951397Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops/TEST-TestShardedTensorElementWiseOps-20221123010445.xml 2022-11-23T01:04:45.4951772Z 2022-11-23T01:04:45.4952258Z ##[endgroup] 2022-11-23T01:04:45.4953161Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_elementwise_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_elementwise_ops_rfx2tptg) 2022-11-23T01:04:45.4953672Z 2022-11-23T01:04:45.4953929Z Running distributed/_shard/sharded_tensor/ops/test_tensor_ops ... [2022-11-23 01:04:45.494740] 2022-11-23T01:04:45.4954481Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:04:45.495128] 2022-11-23T01:04:46.9538975Z 2022-11-23T01:04:46.9539502Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_tensor_ops 2022-11-23T01:04:46.9540679Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_tensor_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_tensor_ops_tmskyrcn) 2022-11-23T01:04:46.9541737Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops 2022-11-23T01:04:46.9541963Z 2022-11-23T01:04:46.9542039Z Running tests... 2022-11-23T01:04:46.9542442Z ---------------------------------------------------------------------- 2022-11-23T01:04:46.9542813Z test_clone (__main__.TestTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:46.9543197Z test_deep_copy (__main__.TestTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:46.9543610Z test_detach (__main__.TestTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:46.9543964Z test_inplace_copy (__main__.TestTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:46.9544314Z test_set_requires_grad (__main__.TestTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:46.9544512Z 2022-11-23T01:04:46.9544717Z ---------------------------------------------------------------------- 2022-11-23T01:04:46.9544955Z Ran 5 tests in 0.003s 2022-11-23T01:04:46.9545067Z 2022-11-23T01:04:46.9545138Z OK (skipped=5) 2022-11-23T01:04:46.9545234Z 2022-11-23T01:04:46.9545318Z Generating XML reports... 2022-11-23T01:04:46.9545891Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops/TEST-TestTensorOps-20221123010446.xml 2022-11-23T01:04:46.9546158Z 2022-11-23T01:04:46.9546385Z ##[endgroup] 2022-11-23T01:04:46.9546934Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_tensor_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_tensor_ops_tmskyrcn) 2022-11-23T01:04:46.9547236Z 2022-11-23T01:04:46.9547471Z Running distributed/_shard/sharding_plan/test_sharding_plan ... [2022-11-23 01:04:46.954177] 2022-11-23T01:04:46.9548030Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharding_plan/test_sharding_plan.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:04:46.954506] 2022-11-23T01:04:48.4307616Z 2022-11-23T01:04:48.4308256Z Expand the folded group to see the log file of distributed/_shard/sharding_plan/test_sharding_plan 2022-11-23T01:04:48.4309530Z ##[group]PRINTING LOG FILE of distributed/_shard/sharding_plan/test_sharding_plan (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharding_plan-test_sharding_plan__1uojwq0) 2022-11-23T01:04:48.4310240Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharding_plan.test_sharding_plan 2022-11-23T01:04:48.4310462Z 2022-11-23T01:04:48.4310537Z Running tests... 2022-11-23T01:04:48.4310898Z ---------------------------------------------------------------------- 2022-11-23T01:04:48.4311248Z test_custom_sharding_planner (__main__.TestShardingPlan) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:48.4311701Z test_reshard_to_ddp_sharding_plan (__main__.TestShardingPlan) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-11-23T01:04:48.4312325Z test_shard_module_sub_process_group (__main__.TestShardingPlan) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:48.4312728Z test_sharding_plan_errors (__main__.TestShardingPlan) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:48.4313177Z test_sharding_plan_simple_megatron (__main__.TestShardingPlan) ... skip: c10d was not compiled with the NCCL backend (0.003s) 2022-11-23T01:04:48.4313396Z 2022-11-23T01:04:48.4313606Z ---------------------------------------------------------------------- 2022-11-23T01:04:48.4313847Z Ran 5 tests in 0.008s 2022-11-23T01:04:48.4313959Z 2022-11-23T01:04:48.4314019Z OK (skipped=5) 2022-11-23T01:04:48.4314125Z 2022-11-23T01:04:48.4314208Z Generating XML reports... 2022-11-23T01:04:48.4314687Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_plan.test_sharding_plan/TEST-TestShardingPlan-20221123010448.xml 2022-11-23T01:04:48.4314961Z 2022-11-23T01:04:48.4315194Z ##[endgroup] 2022-11-23T01:04:48.4315726Z FINISHED PRINTING LOG FILE of distributed/_shard/sharding_plan/test_sharding_plan (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharding_plan-test_sharding_plan__1uojwq0) 2022-11-23T01:04:48.4316030Z 2022-11-23T01:04:48.4316342Z Running distributed/_shard/sharded_tensor/ops/test_matrix_ops ... [2022-11-23 01:04:48.431149] 2022-11-23T01:04:48.4317633Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_matrix_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:04:48.431521] 2022-11-23T01:04:49.9060964Z 2022-11-23T01:04:49.9061580Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_matrix_ops 2022-11-23T01:04:49.9062860Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_matrix_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_matrix_ops_f3v_j4be) 2022-11-23T01:04:49.9063625Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_matrix_ops 2022-11-23T01:04:49.9063847Z 2022-11-23T01:04:49.9063924Z Running tests... 2022-11-23T01:04:49.9064242Z ---------------------------------------------------------------------- 2022-11-23T01:04:49.9064630Z test_sharded_tensor_contiguous (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:49.9065057Z test_sharded_tensor_layer_norm (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:49.9065741Z test_sharded_tensor_layer_norm_error (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:49.9066652Z test_sharded_tensor_masked_fill (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-11-23T01:04:49.9067140Z test_sharded_tensor_masked_fill_error (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:49.9067564Z test_sharded_tensor_softmax (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:49.9068157Z test_sharded_tensor_transpose (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:49.9068597Z test_sharded_tensor_transpose_error (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-11-23T01:04:49.9069043Z test_sharded_tensor_type_as (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:49.9069565Z test_sharded_tensor_view (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:49.9070090Z test_sharded_tensor_view_error (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:49.9070387Z 2022-11-23T01:04:49.9070607Z ---------------------------------------------------------------------- 2022-11-23T01:04:49.9070854Z Ran 11 tests in 0.008s 2022-11-23T01:04:49.9070971Z 2022-11-23T01:04:49.9071034Z OK (skipped=11) 2022-11-23T01:04:49.9071144Z 2022-11-23T01:04:49.9071232Z Generating XML reports... 2022-11-23T01:04:49.9071762Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_matrix_ops/TEST-TestShardedTensorMatrixOps-20221123010449.xml 2022-11-23T01:04:49.9072139Z 2022-11-23T01:04:49.9072379Z ##[endgroup] 2022-11-23T01:04:49.9072894Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_matrix_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_matrix_ops_f3v_j4be) 2022-11-23T01:04:49.9073191Z 2022-11-23T01:04:49.9073398Z Running distributed/test_dynamo_distributed ... [2022-11-23 01:04:49.906501] 2022-11-23T01:04:49.9073937Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_dynamo_distributed.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:04:49.906878] 2022-11-23T01:04:51.5360087Z 2022-11-23T01:04:51.5361076Z Expand the folded group to see the log file of distributed/test_dynamo_distributed 2022-11-23T01:04:51.5362031Z ##[group]PRINTING LOG FILE of distributed/test_dynamo_distributed (/var/lib/jenkins/workspace/test/test-reports/distributed-test_dynamo_distributed_afatw3g8) 2022-11-23T01:04:51.5362302Z 2022-11-23T01:04:51.5362376Z Running tests... 2022-11-23T01:04:51.5362786Z ---------------------------------------------------------------------- 2022-11-23T01:04:51.5363067Z test_aot_autograd (__main__.TestDistributed) 2022-11-23T01:04:51.5363557Z Explicitly check AotAutograd family of compilers work, ... Test results will be stored in test-reports/python-unittest/distributed.test_dynamo_distributed 2022-11-23T01:04:51.5363947Z skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:51.5364187Z test_custom_layer (__main__.TestDistributed) 2022-11-23T01:04:51.5364690Z Just ensures that the appropriate number of splits happen (based on ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:51.5365162Z test_ddp_baseline_aot_eager (__main__.TestDistributed) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-11-23T01:04:51.5365534Z test_ddp_baseline_inductor (__main__.TestDistributed) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-11-23T01:04:51.5365914Z test_empty_graph_inductor (__main__.TestDistributed) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-11-23T01:04:51.5366219Z test_graph_split (__main__.TestDistributed) 2022-11-23T01:04:51.5366551Z Just ensures that the appropriate number of splits happen (based on ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:51.5366870Z test_graph_split_inductor (__main__.TestDistributed) 2022-11-23T01:04:51.5367183Z Same as above, but using inductor backend. ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:51.5367616Z test_ignored_parameters (__main__.TestDistributed) 2022-11-23T01:04:51.5368392Z Verifies ddp graph-split logic ignores parameters marked to ignore on DDP module. ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:51.5368719Z test_no_split (__main__.TestDistributed) 2022-11-23T01:04:51.5369054Z Ensures the DDPOptimizer returns a correct, compiled module without ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:51.5369491Z test_ddp_baseline_aot_eager_multiprocess (__main__.TestDistributedMultiProc) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-11-23T01:04:51.5369906Z test_fsdp_aot_eager (__main__.TestDistributedMultiProc) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:51.5370372Z test_fsdp_inductor (__main__.TestDistributedMultiProc) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:51.5370768Z test_hf_bert_ddp (__main__.TestDistributedMultiProc) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:51.5371162Z test_hf_bert_fsdp (__main__.TestDistributedMultiProc) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:04:51.5371370Z 2022-11-23T01:04:51.5371561Z ---------------------------------------------------------------------- 2022-11-23T01:04:51.5371806Z Ran 14 tests in 0.010s 2022-11-23T01:04:51.5371922Z 2022-11-23T01:04:51.5371996Z OK (skipped=14) 2022-11-23T01:04:51.5372104Z 2022-11-23T01:04:51.5372188Z Generating XML reports... 2022-11-23T01:04:51.5372625Z Generated XML report: test-reports/python-unittest/distributed.test_dynamo_distributed/TEST-TestDistributed-20221123010451.xml 2022-11-23T01:04:51.5373220Z Generated XML report: test-reports/python-unittest/distributed.test_dynamo_distributed/TEST-TestDistributedMultiProc-20221123010451.xml 2022-11-23T01:04:51.5373495Z 2022-11-23T01:04:51.5373751Z ##[endgroup] 2022-11-23T01:04:51.5374195Z FINISHED PRINTING LOG FILE of distributed/test_dynamo_distributed (/var/lib/jenkins/workspace/test/test-reports/distributed-test_dynamo_distributed_afatw3g8) 2022-11-23T01:04:51.5374462Z 2022-11-23T01:04:51.5374681Z Running distributed/_composable/test_checkpoint ... [2022-11-23 01:04:51.536333] 2022-11-23T01:04:51.5375219Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_composable/test_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:04:51.536667] 2022-11-23T01:04:53.3677917Z 2022-11-23T01:04:53.3678632Z Expand the folded group to see the log file of distributed/_composable/test_checkpoint 2022-11-23T01:04:53.3679426Z ##[group]PRINTING LOG FILE of distributed/_composable/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_checkpoint_yrmmm16f) 2022-11-23T01:04:53.3680114Z Test results will be stored in test-reports/python-unittest/distributed._composable.test_checkpoint 2022-11-23T01:04:53.3680345Z 2022-11-23T01:04:53.3680420Z Running tests... 2022-11-23T01:04:53.3680905Z ---------------------------------------------------------------------- 2022-11-23T01:04:53.3681218Z test_tensor_only_cpu (__main__.TestCheckpoint) ... ok (0.015s) 2022-11-23T01:04:53.3681558Z test_tensor_only_gpu (__main__.TestCheckpoint) ... skip: no cuda (0.000s) 2022-11-23T01:04:53.3681730Z 2022-11-23T01:04:53.3681937Z ---------------------------------------------------------------------- 2022-11-23T01:04:53.3682306Z Ran 2 tests in 0.015s 2022-11-23T01:04:53.3682420Z 2022-11-23T01:04:53.3682494Z OK (skipped=1) 2022-11-23T01:04:53.3682601Z 2022-11-23T01:04:53.3682685Z Generating XML reports... 2022-11-23T01:04:53.3683132Z Generated XML report: test-reports/python-unittest/distributed._composable.test_checkpoint/TEST-TestCheckpoint-20221123010453.xml 2022-11-23T01:04:53.3683388Z 2022-11-23T01:04:53.3683613Z ##[endgroup] 2022-11-23T01:04:53.3684087Z FINISHED PRINTING LOG FILE of distributed/_composable/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_checkpoint_yrmmm16f) 2022-11-23T01:04:53.3684343Z 2022-11-23T01:04:53.3684731Z Running distributed/elastic/metrics/api_test ... [2022-11-23 01:04:53.368045] 2022-11-23T01:04:53.3685617Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/metrics/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:04:53.368410] 2022-11-23T01:04:55.1727227Z 2022-11-23T01:04:55.1727709Z Expand the folded group to see the log file of distributed/elastic/metrics/api_test 2022-11-23T01:04:55.1728540Z ##[group]PRINTING LOG FILE of distributed/elastic/metrics/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-metrics-api_test_9qo4mh57) 2022-11-23T01:04:55.1729219Z Test results will be stored in test-reports/python-unittest/distributed.elastic.metrics.api_test 2022-11-23T01:04:55.1729672Z 2022-11-23T01:04:55.1729750Z Running tests... 2022-11-23T01:04:55.1730072Z ---------------------------------------------------------------------- 2022-11-23T01:04:55.1730396Z test_get_metric_name (__main__.MetricsApiTest) ... ok (0.259s) 2022-11-23T01:04:55.1730683Z test_inheritance (__main__.MetricsApiTest) ... ok (0.001s) 2022-11-23T01:04:55.1731024Z test_profile (__main__.MetricsApiTest) ... ok (0.002s) 2022-11-23T01:04:55.1731236Z 2022-11-23T01:04:55.1731440Z ---------------------------------------------------------------------- 2022-11-23T01:04:55.1731716Z Ran 3 tests in 0.262s 2022-11-23T01:04:55.1731831Z 2022-11-23T01:04:55.1731908Z OK 2022-11-23T01:04:55.1732053Z 2022-11-23T01:04:55.1732126Z Generating XML reports... 2022-11-23T01:04:55.1732591Z Generated XML report: test-reports/python-unittest/distributed.elastic.metrics.api_test/TEST-MetricsApiTest-20221123010454.xml 2022-11-23T01:04:55.1732909Z 2022-11-23T01:04:55.1733142Z ##[endgroup] 2022-11-23T01:04:55.1733658Z FINISHED PRINTING LOG FILE of distributed/elastic/metrics/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-metrics-api_test_9qo4mh57) 2022-11-23T01:04:55.1733934Z 2022-11-23T01:04:55.1734175Z Running distributed/nn/jit/test_instantiator ... [2022-11-23 01:04:55.173046] 2022-11-23T01:04:55.1735731Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/nn/jit/test_instantiator.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:04:55.173374] 2022-11-23T01:04:56.9818246Z 2022-11-23T01:04:56.9818826Z Expand the folded group to see the log file of distributed/nn/jit/test_instantiator 2022-11-23T01:04:56.9820006Z ##[group]PRINTING LOG FILE of distributed/nn/jit/test_instantiator (/var/lib/jenkins/workspace/test/test-reports/distributed-nn-jit-test_instantiator_mnqv3rvb) 2022-11-23T01:04:56.9821123Z Test results will be stored in test-reports/python-unittest/distributed.nn.jit.test_instantiator 2022-11-23T01:04:56.9821363Z 2022-11-23T01:04:56.9821452Z Running tests... 2022-11-23T01:04:56.9821763Z ---------------------------------------------------------------------- 2022-11-23T01:04:56.9822076Z test_get_arg_return_types_from_interface (__main__.TestInstantiator) ... ok (0.263s) 2022-11-23T01:04:56.9822462Z test_instantiate_non_scripted_remote_module_template (__main__.TestInstantiator) ... ok (0.002s) 2022-11-23T01:04:56.9822841Z test_instantiate_scripted_remote_module_template (__main__.TestInstantiator) ... ok (0.010s) 2022-11-23T01:04:56.9823076Z 2022-11-23T01:04:56.9823309Z ---------------------------------------------------------------------- 2022-11-23T01:04:56.9823580Z Ran 3 tests in 0.276s 2022-11-23T01:04:56.9823695Z 2022-11-23T01:04:56.9823757Z OK 2022-11-23T01:04:56.9823896Z 2022-11-23T01:04:56.9823970Z Generating XML reports... 2022-11-23T01:04:56.9824442Z Generated XML report: test-reports/python-unittest/distributed.nn.jit.test_instantiator/TEST-TestInstantiator-20221123010456.xml 2022-11-23T01:04:56.9824760Z 2022-11-23T01:04:56.9824985Z ##[endgroup] 2022-11-23T01:04:56.9825490Z FINISHED PRINTING LOG FILE of distributed/nn/jit/test_instantiator (/var/lib/jenkins/workspace/test/test-reports/distributed-nn-jit-test_instantiator_mnqv3rvb) 2022-11-23T01:04:56.9825758Z 2022-11-23T01:04:56.9826176Z Running distributed/checkpoint/test_utils ... [2022-11-23 01:04:56.982175] 2022-11-23T01:04:56.9827024Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:04:56.982521] 2022-11-23T01:04:58.8204980Z 2022-11-23T01:04:58.8205560Z Expand the folded group to see the log file of distributed/checkpoint/test_utils 2022-11-23T01:04:58.8206675Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_utils_dc7435i9) 2022-11-23T01:04:58.8207300Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_utils 2022-11-23T01:04:58.8207724Z 2022-11-23T01:04:58.8207798Z Running tests... 2022-11-23T01:04:58.8208099Z ---------------------------------------------------------------------- 2022-11-23T01:04:58.8208390Z test_flat_data (__main__.TestMedatadaIndex) ... ok (0.274s) 2022-11-23T01:04:58.8208703Z test_index_hint_ignored_on_equals (__main__.TestMedatadaIndex) ... ok (0.001s) 2022-11-23T01:04:58.8209031Z test_index_hint_ignored_on_hash (__main__.TestMedatadaIndex) ... ok (0.001s) 2022-11-23T01:04:58.8209313Z test_init_convert_offset (__main__.TestMedatadaIndex) ... ok (0.001s) 2022-11-23T01:04:58.8209604Z test_sharded_tensor_lookup (__main__.TestMedatadaIndex) ... ok (0.003s) 2022-11-23T01:04:58.8209770Z 2022-11-23T01:04:58.8209970Z ---------------------------------------------------------------------- 2022-11-23T01:04:58.8210196Z Ran 5 tests in 0.279s 2022-11-23T01:04:58.8210312Z 2022-11-23T01:04:58.8210372Z OK 2022-11-23T01:04:58.8210463Z 2022-11-23T01:04:58.8210547Z Generating XML reports... 2022-11-23T01:04:58.8211014Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_utils/TEST-TestMedatadaIndex-20221123010458.xml 2022-11-23T01:04:58.8211266Z 2022-11-23T01:04:58.8211486Z ##[endgroup] 2022-11-23T01:04:58.8211941Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_utils_dc7435i9) 2022-11-23T01:04:58.8212251Z 2022-11-23T01:04:58.8212461Z Running distributed/checkpoint/test_planner ... [2022-11-23 01:04:58.820809] 2022-11-23T01:04:58.8213292Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_planner.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:04:58.821165] 2022-11-23T01:05:00.6765628Z 2022-11-23T01:05:00.6766159Z Expand the folded group to see the log file of distributed/checkpoint/test_planner 2022-11-23T01:05:00.6767246Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_planner (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_planner_2ckioh8w) 2022-11-23T01:05:00.6768200Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_planner 2022-11-23T01:05:00.6768564Z 2022-11-23T01:05:00.6768679Z Running tests... 2022-11-23T01:05:00.6769182Z ---------------------------------------------------------------------- 2022-11-23T01:05:00.6769656Z test_global_plan (__main__.TestSavePlan) ... ok (0.279s) 2022-11-23T01:05:00.6770102Z test_load_with_resharding (__main__.TestSavePlan) ... ok (0.004s) 2022-11-23T01:05:00.6770588Z test_load_with_world_size_diff_by_one (__main__.TestSavePlan) ... ok (0.003s) 2022-11-23T01:05:00.6771076Z test_local_load_plan (__main__.TestSavePlan) ... ok (0.003s) 2022-11-23T01:05:00.6771488Z test_local_plan (__main__.TestSavePlan) ... ok (0.003s) 2022-11-23T01:05:00.6771723Z 2022-11-23T01:05:00.6772046Z ---------------------------------------------------------------------- 2022-11-23T01:05:00.6772426Z Ran 5 tests in 0.292s 2022-11-23T01:05:00.6772600Z 2022-11-23T01:05:00.6772705Z OK 2022-11-23T01:05:00.6772845Z 2022-11-23T01:05:00.6772978Z Generating XML reports... 2022-11-23T01:05:00.6773652Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_planner/TEST-TestSavePlan-20221123010500.xml 2022-11-23T01:05:00.6774079Z 2022-11-23T01:05:00.6774521Z ##[endgroup] 2022-11-23T01:05:00.6775659Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_planner (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_planner_2ckioh8w) 2022-11-23T01:05:00.6776048Z 2022-11-23T01:05:00.6776297Z Running distributed/checkpoint/test_file_system_checkpoint ... [2022-11-23 01:05:00.676913] 2022-11-23T01:05:00.6776849Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:05:00.677285] 2022-11-23T01:05:02.5728746Z 2022-11-23T01:05:02.5729338Z Expand the folded group to see the log file of distributed/checkpoint/test_file_system_checkpoint 2022-11-23T01:05:02.5730754Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_file_system_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_file_system_checkpoint_fx06xi0w) 2022-11-23T01:05:02.5731480Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint 2022-11-23T01:05:02.5731734Z 2022-11-23T01:05:02.5731812Z Running tests... 2022-11-23T01:05:02.5732127Z ---------------------------------------------------------------------- 2022-11-23T01:05:02.5732588Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:05:02.5733124Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-11-23T01:05:02.5733573Z test_save_load_bytes (__main__.TestDistributedReshardOnLoad) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:05:02.5734064Z test_switch_between_sharded_tensor_to_tensor (__main__.TestDistributedReshardOnLoad) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-11-23T01:05:02.5734473Z test_read_write_only_tensor (__main__.TestDistributedStateDictSaveLoad) ... ok (0.295s) 2022-11-23T01:05:02.5734917Z test_read_write_shard_tensor (__main__.TestDistributedStateDictSaveLoadWithSharedTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:05:02.5735198Z 2022-11-23T01:05:02.5735415Z ---------------------------------------------------------------------- 2022-11-23T01:05:02.5735684Z Ran 6 tests in 0.302s 2022-11-23T01:05:02.5735833Z 2022-11-23T01:05:02.5735939Z OK (skipped=5) 2022-11-23T01:05:02.5736051Z 2022-11-23T01:05:02.5736138Z Generating XML reports... 2022-11-23T01:05:02.5736784Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoad-20221123010501.xml 2022-11-23T01:05:02.5737557Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedReshardOnLoad-20221123010501.xml 2022-11-23T01:05:02.5738337Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20221123010501.xml 2022-11-23T01:05:02.5738710Z 2022-11-23T01:05:02.5738930Z ##[endgroup] 2022-11-23T01:05:02.5739447Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_file_system_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_file_system_checkpoint_fx06xi0w) 2022-11-23T01:05:02.5739749Z 2022-11-23T01:05:02.5739968Z Running distributed/fsdp/test_checkpoint_wrapper ... [2022-11-23 01:05:02.573186] 2022-11-23T01:05:02.5740509Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_checkpoint_wrapper.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:05:02.573519] 2022-11-23T01:05:04.4827349Z 2022-11-23T01:05:04.4827945Z Expand the folded group to see the log file of distributed/fsdp/test_checkpoint_wrapper 2022-11-23T01:05:04.4829000Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_checkpoint_wrapper (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_checkpoint_wrapper_tk8824qu) 2022-11-23T01:05:04.4829846Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper 2022-11-23T01:05:04.4830069Z 2022-11-23T01:05:04.4830143Z Running tests... 2022-11-23T01:05:04.4830456Z ---------------------------------------------------------------------- 2022-11-23T01:05:04.4830776Z test_apply_activation_checkpointing (__main__.CheckpointWrapperTest) 2022-11-23T01:05:04.4831111Z Ensures that `apply_activation_checkpointing` can be used ... ok (0.325s) 2022-11-23T01:05:04.4831566Z test_checkpoint_wrapper_cpu_offload (__main__.CheckpointWrapperTest) ... skip: Test requires CUDA (0.001s) 2022-11-23T01:05:04.4832009Z test_checkpoint_wrapper_kwarg_support (__main__.CheckpointWrapperTest) ... ok (0.007s) 2022-11-23T01:05:04.4832390Z test_checkpoint_wrapper_parity (__main__.CheckpointWrapperTest) 2022-11-23T01:05:04.4832716Z Tests that using checkpoint_wrapper or the functional ... skip: Test requires CUDA (0.002s) 2022-11-23T01:05:04.4833058Z test_forward_missing_attributes (__main__.CheckpointWrapperTest) ... ok (0.001s) 2022-11-23T01:05:04.4833360Z test_fqn (__main__.CheckpointWrapperTest) ... ok (0.001s) 2022-11-23T01:05:04.4833657Z test_load_activation_checkpointed_module (__main__.CheckpointWrapperTest) ... ok (0.002s) 2022-11-23T01:05:04.4833851Z 2022-11-23T01:05:04.4834059Z ---------------------------------------------------------------------- 2022-11-23T01:05:04.4834304Z Ran 7 tests in 0.340s 2022-11-23T01:05:04.4834417Z 2022-11-23T01:05:04.4834477Z OK (skipped=2) 2022-11-23T01:05:04.4834584Z 2022-11-23T01:05:04.4834706Z Generating XML reports... 2022-11-23T01:05:04.4835345Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper/TEST-CheckpointWrapperTest-20221123010503.xml 2022-11-23T01:05:04.4835630Z 2022-11-23T01:05:04.4835932Z ##[endgroup] 2022-11-23T01:05:04.4836411Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_checkpoint_wrapper (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_checkpoint_wrapper_tk8824qu) 2022-11-23T01:05:04.4836741Z 2022-11-23T01:05:04.4836952Z Running distributed/elastic/utils/util_test ... [2022-11-23 01:05:04.483054] 2022-11-23T01:05:04.4837518Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/utils/util_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:05:04.483424] 2022-11-23T01:05:06.3949314Z 2022-11-23T01:05:06.3949923Z Expand the folded group to see the log file of distributed/elastic/utils/util_test 2022-11-23T01:05:06.3950915Z ##[group]PRINTING LOG FILE of distributed/elastic/utils/util_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-util_test_ph94wk36) 2022-11-23T01:05:06.3951605Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.util_test 2022-11-23T01:05:06.3951824Z 2022-11-23T01:05:06.3951885Z Running tests... 2022-11-23T01:05:06.3952259Z ---------------------------------------------------------------------- 2022-11-23T01:05:06.3952564Z test_get_all_rank_0 (__main__.StoreUtilTest) ... ok (0.263s) 2022-11-23T01:05:06.3952850Z test_get_all_rank_n (__main__.StoreUtilTest) ... ok (0.002s) 2022-11-23T01:05:06.3953115Z test_synchronize (__main__.StoreUtilTest) ... ok (0.002s) 2022-11-23T01:05:06.3953366Z test_get_logger (__main__.UtilTest) ... ok (0.085s) 2022-11-23T01:05:06.3953735Z test_get_logger_custom_name (__main__.UtilTest) ... ok (0.001s) 2022-11-23T01:05:06.3954038Z test_get_logger_different (__main__.UtilTest) ... ok (0.001s) 2022-11-23T01:05:06.3954299Z test_get_logger_none (__main__.UtilTest) ... ok (0.001s) 2022-11-23T01:05:06.3954443Z 2022-11-23T01:05:06.3954643Z ---------------------------------------------------------------------- 2022-11-23T01:05:06.3954933Z Ran 7 tests in 0.355s 2022-11-23T01:05:06.3955046Z 2022-11-23T01:05:06.3955106Z OK 2022-11-23T01:05:06.3955198Z 2022-11-23T01:05:06.3955282Z Generating XML reports... 2022-11-23T01:05:06.3955999Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-StoreUtilTest-20221123010505.xml 2022-11-23T01:05:06.3956646Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-UtilTest-20221123010505.xml 2022-11-23T01:05:06.3957032Z 2022-11-23T01:05:06.3957304Z ##[endgroup] 2022-11-23T01:05:06.3957802Z FINISHED PRINTING LOG FILE of distributed/elastic/utils/util_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-util_test_ph94wk36) 2022-11-23T01:05:06.3958076Z 2022-11-23T01:05:06.3958284Z Running distributed/fsdp/test_fsdp_overlap ... [2022-11-23 01:05:06.395241] 2022-11-23T01:05:06.3958862Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_overlap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:05:06.395579] 2022-11-23T01:05:09.9056448Z 2022-11-23T01:05:09.9057040Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_overlap 2022-11-23T01:05:09.9058093Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_overlap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_overlap_v9x0_dan) 2022-11-23T01:05:09.9058731Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap 2022-11-23T01:05:09.9058937Z 2022-11-23T01:05:09.9059015Z Running tests... 2022-11-23T01:05:09.9059321Z ---------------------------------------------------------------------- 2022-11-23T01:05:09.9059769Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1220 2022-11-23T01:05:09.9060300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:09.9060765Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:09.9061547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:09.9061888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:09.9062228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:05:09.9062727Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T01:05:09.9063175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:05:09.9063545Z dist init r=0, world=1 2022-11-23T01:05:09.9063797Z skip: Need at least 2 CUDA devices (1.873s) 2022-11-23T01:05:09.9065061Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/71183 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s) 2022-11-23T01:05:09.9065534Z 2022-11-23T01:05:09.9065734Z ---------------------------------------------------------------------- 2022-11-23T01:05:09.9065976Z Ran 2 tests in 1.874s 2022-11-23T01:05:09.9066077Z 2022-11-23T01:05:09.9066148Z OK (skipped=2) 2022-11-23T01:05:09.9066253Z 2022-11-23T01:05:09.9066337Z Generating XML reports... 2022-11-23T01:05:09.9066829Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20221123010507.xml 2022-11-23T01:05:09.9067479Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20221123010507.xml 2022-11-23T01:05:09.9067769Z 2022-11-23T01:05:09.9068013Z ##[endgroup] 2022-11-23T01:05:09.9068471Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_overlap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_overlap_v9x0_dan) 2022-11-23T01:05:09.9068736Z 2022-11-23T01:05:09.9069164Z Running distributed/fsdp/test_fsdp_uneven ... [2022-11-23 01:05:09.905879] 2022-11-23T01:05:09.9069671Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_uneven.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:05:09.906244] 2022-11-23T01:05:13.6290909Z 2022-11-23T01:05:13.6291462Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_uneven 2022-11-23T01:05:13.6292546Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_uneven (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_uneven_xj8s52my) 2022-11-23T01:05:13.6293189Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven 2022-11-23T01:05:13.6293608Z 2022-11-23T01:05:13.6293684Z Running tests... 2022-11-23T01:05:13.6293997Z ---------------------------------------------------------------------- 2022-11-23T01:05:13.6294275Z test_one_iteration (__main__.TestUnevenParamShard) 2022-11-23T01:05:13.6294650Z Test FSDP with uneven divide of parameter shards. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1245 2022-11-23T01:05:13.6295037Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1246 2022-11-23T01:05:13.6295363Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1247 2022-11-23T01:05:13.6295673Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1248 2022-11-23T01:05:13.6296136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:13.6296469Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:13.6297154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:13.6297567Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:13.6297906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:05:13.6298386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:13.6298702Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:13.6299131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:13.6299479Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:13.6299801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:05:13.6300264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:13.6300667Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:13.6301205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:13.6301560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:13.6301894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:05:13.6302361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:13.6302687Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:13.6303097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:13.6303442Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:13.6303778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:05:13.6304260Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:13.6304869Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:13.6305391Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:13.6305914Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:13.6306287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:05:13.6306638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:05:13.6306979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:05:13.6307392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:05:13.6307640Z dist init r=1, world=4 2022-11-23T01:05:13.6307822Z dist init r=2, world=4 2022-11-23T01:05:13.6308001Z dist init r=0, world=4 2022-11-23T01:05:13.6308168Z dist init r=3, world=4 2022-11-23T01:05:13.6308375Z skip: Need at least 2 CUDA devices (2.086s) 2022-11-23T01:05:13.6308515Z 2022-11-23T01:05:13.6308717Z ---------------------------------------------------------------------- 2022-11-23T01:05:13.6308947Z Ran 1 test in 2.086s 2022-11-23T01:05:13.6309060Z 2022-11-23T01:05:13.6309130Z OK (skipped=1) 2022-11-23T01:05:13.6309236Z 2022-11-23T01:05:13.6309325Z Generating XML reports... 2022-11-23T01:05:13.6309782Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven/TEST-TestUnevenParamShard-20221123010511.xml 2022-11-23T01:05:13.6310037Z 2022-11-23T01:05:13.6310311Z ##[endgroup] 2022-11-23T01:05:13.6310764Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_uneven (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_uneven_xj8s52my) 2022-11-23T01:05:13.6311024Z 2022-11-23T01:05:13.6311236Z Running distributed/fsdp/test_fsdp_pure_fp16 ... [2022-11-23 01:05:13.629497] 2022-11-23T01:05:13.6311747Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_pure_fp16.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:05:13.629950] 2022-11-23T01:05:17.3459655Z 2022-11-23T01:05:17.3460224Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_pure_fp16 2022-11-23T01:05:17.3461361Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_pure_fp16 (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_pure_fp16_bo2reasr) 2022-11-23T01:05:17.3462306Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_pure_fp16 2022-11-23T01:05:17.3462692Z 2022-11-23T01:05:17.3462829Z Running tests... 2022-11-23T01:05:17.3463295Z ---------------------------------------------------------------------- 2022-11-23T01:05:17.3463610Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T01:05:17.3464696Z Tests pure FP16 training, including when the parameter's dtype is ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/73315 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.265s) 2022-11-23T01:05:17.3465688Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=True) (__main__.TestPureFP16) 2022-11-23T01:05:17.3466753Z Tests pure FP16 training, including when the parameter's dtype is ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1312 2022-11-23T01:05:17.3467434Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1313 2022-11-23T01:05:17.3468004Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1314 2022-11-23T01:05:17.3468591Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1315 2022-11-23T01:05:17.3469373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:17.3469699Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:17.3470133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:17.3470527Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:17.3470889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:05:17.3471448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:17.3471856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:17.3472368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:17.3472700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:17.3473040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:05:17.3473511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:17.3473838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:17.3474250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:17.3474593Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:17.3474955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:05:17.3475424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:17.3475737Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:17.3476162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:17.3476505Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:17.3476835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:05:17.3477314Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:17.3477845Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:17.3478366Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:17.3478883Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:17.3479256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:05:17.3479601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:05:17.3479947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:05:17.3480271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:05:17.3480529Z dist init r=3, world=4 2022-11-23T01:05:17.3480935Z dist init r=1, world=4 2022-11-23T01:05:17.3481101Z dist init r=2, world=4 2022-11-23T01:05:17.3481284Z dist init r=0, world=4 2022-11-23T01:05:17.3481495Z skip: Need at least 2 CUDA devices (1.824s) 2022-11-23T01:05:17.3481635Z 2022-11-23T01:05:17.3481839Z ---------------------------------------------------------------------- 2022-11-23T01:05:17.3482073Z Ran 2 tests in 2.089s 2022-11-23T01:05:17.3482186Z 2022-11-23T01:05:17.3482258Z OK (skipped=2) 2022-11-23T01:05:17.3482364Z 2022-11-23T01:05:17.3482450Z Generating XML reports... 2022-11-23T01:05:17.3482948Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_pure_fp16/TEST-TestPureFP16-20221123010514.xml 2022-11-23T01:05:17.3483204Z 2022-11-23T01:05:17.3483481Z ##[endgroup] 2022-11-23T01:05:17.3483946Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_pure_fp16 (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_pure_fp16_bo2reasr) 2022-11-23T01:05:17.3484211Z 2022-11-23T01:05:17.3484435Z Running distributed/_tensor/parallel/test_view_sharding_dim_change ... [2022-11-23 01:05:17.346575] 2022-11-23T01:05:17.3485018Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/parallel/test_view_sharding_dim_change.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:05:17.346967] 2022-11-23T01:05:21.1704152Z 2022-11-23T01:05:21.1704750Z Expand the folded group to see the log file of distributed/_tensor/parallel/test_view_sharding_dim_change 2022-11-23T01:05:21.1706088Z ##[group]PRINTING LOG FILE of distributed/_tensor/parallel/test_view_sharding_dim_change (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-parallel-test_view_sharding_dim_change_fue5_scx) 2022-11-23T01:05:21.1707279Z Test results will be stored in test-reports/python-unittest/distributed._tensor.parallel.test_view_sharding_dim_change 2022-11-23T01:05:21.1707720Z 2022-11-23T01:05:21.1707836Z Running tests... 2022-11-23T01:05:21.1708321Z ---------------------------------------------------------------------- 2022-11-23T01:05:21.1708941Z test_view_with_sharding_dim_change (__main__.TPViewShardingDimChangeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1379 2022-11-23T01:05:21.1709640Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1380 2022-11-23T01:05:21.1710175Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1381 2022-11-23T01:05:21.1710655Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1382 2022-11-23T01:05:21.1711482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:21.1712098Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:21.1712775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:21.1713309Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:21.1713849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:05:21.1714607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:21.1715082Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:21.1715657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:21.1716126Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:21.1716576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:05:21.1717212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:21.1717700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:21.1718334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:21.1718872Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:21.1719357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:05:21.1720050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:21.1720540Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:21.1721699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:21.1722188Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:21.1722635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:05:21.1723121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:05:21.1723610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:05:21.1724109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:05:21.1724686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:05:21.1725397Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:21.1726119Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:21.1726860Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:21.1727619Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:21.1728031Z ok (2.176s) 2022-11-23T01:05:21.1728164Z 2022-11-23T01:05:21.1728468Z ---------------------------------------------------------------------- 2022-11-23T01:05:21.1728814Z Ran 1 test in 2.177s 2022-11-23T01:05:21.1728981Z 2022-11-23T01:05:21.1729070Z OK 2022-11-23T01:05:21.1729231Z 2022-11-23T01:05:21.1729336Z Generating XML reports... 2022-11-23T01:05:21.1730117Z Generated XML report: test-reports/python-unittest/distributed._tensor.parallel.test_view_sharding_dim_change/TEST-TPViewShardingDimChangeTest-20221123010518.xml 2022-11-23T01:05:21.1730565Z 2022-11-23T01:05:21.1730945Z ##[endgroup] 2022-11-23T01:05:21.1731714Z FINISHED PRINTING LOG FILE of distributed/_tensor/parallel/test_view_sharding_dim_change (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-parallel-test_view_sharding_dim_change_fue5_scx) 2022-11-23T01:05:21.1732153Z 2022-11-23T01:05:21.1732425Z Running distributed/fsdp/test_fsdp_fx ... [2022-11-23 01:05:21.170820] 2022-11-23T01:05:21.1733149Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_fx.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:05:21.171256] 2022-11-23T01:05:24.8723844Z 2022-11-23T01:05:24.8724462Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_fx 2022-11-23T01:05:24.8725692Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_fx (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_fx_06vx_kp_) 2022-11-23T01:05:24.8726694Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_fx 2022-11-23T01:05:24.8727014Z 2022-11-23T01:05:24.8727133Z Running tests... 2022-11-23T01:05:24.8727580Z ---------------------------------------------------------------------- 2022-11-23T01:05:24.8728048Z test_symbolic_tracing_outputs (__main__.TestSymbolicTracing) 2022-11-23T01:05:24.8728735Z test ``execution_info.module_forward_order`` and ``execution_info.module_to_execution_infos`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1446 2022-11-23T01:05:24.8729416Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1447 2022-11-23T01:05:24.8729928Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1448 2022-11-23T01:05:24.8730463Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1449 2022-11-23T01:05:24.8731231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:24.8731982Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:24.8732740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:24.8733274Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:24.8733753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:05:24.8734394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:24.8734814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:24.8735542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:24.8736024Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:24.8736500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:05:24.8737189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:24.8737703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:24.8738408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:24.8738924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:24.8739411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:05:24.8740097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:24.8740553Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:24.8741147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:24.8741596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:24.8742057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:05:24.8742747Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:24.8743453Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:24.8744183Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:24.8744928Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:24.8745497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:05:24.8745991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:05:24.8746499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:05:24.8747011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:05:24.8747378Z dist init r=0, world=4 2022-11-23T01:05:24.8747629Z dist init r=3, world=4 2022-11-23T01:05:24.8747887Z dist init r=1, world=4 2022-11-23T01:05:24.8748134Z dist init r=2, world=4 2022-11-23T01:05:24.8748358Z ok (2.085s) 2022-11-23T01:05:24.8748499Z 2022-11-23T01:05:24.8748793Z ---------------------------------------------------------------------- 2022-11-23T01:05:24.8749141Z Ran 1 test in 2.086s 2022-11-23T01:05:24.8749293Z 2022-11-23T01:05:24.8749368Z OK 2022-11-23T01:05:24.8749502Z 2022-11-23T01:05:24.8749631Z Generating XML reports... 2022-11-23T01:05:24.8750304Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_fx/TEST-TestSymbolicTracing-20221123010522.xml 2022-11-23T01:05:24.8750763Z 2022-11-23T01:05:24.8751137Z ##[endgroup] 2022-11-23T01:05:24.8751769Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_fx (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_fx_06vx_kp_) 2022-11-23T01:05:24.8752216Z 2022-11-23T01:05:24.8752509Z Running distributed/test_c10d_error_logger ... [2022-11-23 01:05:24.872721] 2022-11-23T01:05:24.8753273Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_error_logger.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:05:24.873149] 2022-11-23T01:05:29.8545409Z 2022-11-23T01:05:29.8546005Z Expand the folded group to see the log file of distributed/test_c10d_error_logger 2022-11-23T01:05:29.8547842Z ##[group]PRINTING LOG FILE of distributed/test_c10d_error_logger (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_error_logger_4qmgkhs5) 2022-11-23T01:05:29.8548911Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_error_logger 2022-11-23T01:05:29.8549293Z 2022-11-23T01:05:29.8549431Z Running tests... 2022-11-23T01:05:29.8549823Z ---------------------------------------------------------------------- 2022-11-23T01:05:29.8550300Z test_exception_handler_with_dist (__main__.C10dErrorLoggerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1513 2022-11-23T01:05:29.8550839Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1514 2022-11-23T01:05:29.8551706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:29.8552175Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:29.8552643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:29.8553071Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:29.8553611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:05:29.8554341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:29.8554961Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:29.8555652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:29.8556059Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:29.8556452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:05:29.8556806Z skip: Need at least 2 CUDA devices (1.777s) 2022-11-23T01:05:29.8557179Z test_get_or_create_logger (__main__.C10dErrorLoggerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1535 2022-11-23T01:05:29.8557635Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1536 2022-11-23T01:05:29.8558180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:29.8558568Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:29.8559015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:29.8559429Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:29.8559830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:05:29.8560352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:29.8560909Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:29.8561412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:29.8561927Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:29.8562301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:05:29.8562625Z ok (1.612s) 2022-11-23T01:05:29.8562758Z 2022-11-23T01:05:29.8562990Z ---------------------------------------------------------------------- 2022-11-23T01:05:29.8563294Z Ran 2 tests in 3.390s 2022-11-23T01:05:29.8563397Z 2022-11-23T01:05:29.8563499Z OK (skipped=1) 2022-11-23T01:05:29.8563660Z 2022-11-23T01:05:29.8563788Z Generating XML reports... 2022-11-23T01:05:29.8564300Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_error_logger/TEST-C10dErrorLoggerTest-20221123010526.xml 2022-11-23T01:05:29.8564644Z 2022-11-23T01:05:29.8564937Z ##[endgroup] 2022-11-23T01:05:29.8565449Z FINISHED PRINTING LOG FILE of distributed/test_c10d_error_logger (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_error_logger_4qmgkhs5) 2022-11-23T01:05:29.8565766Z 2022-11-23T01:05:29.8566013Z Running distributed/fsdp/test_fsdp_memory ... [2022-11-23 01:05:29.854899] 2022-11-23T01:05:29.8566590Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_memory.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:05:29.855260] 2022-11-23T01:05:34.9613066Z 2022-11-23T01:05:34.9614002Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_memory 2022-11-23T01:05:34.9652593Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_memory (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_memory_o90wlh7b) 2022-11-23T01:05:34.9653273Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_memory 2022-11-23T01:05:34.9653491Z 2022-11-23T01:05:34.9653557Z Running tests... 2022-11-23T01:05:34.9653853Z ---------------------------------------------------------------------- 2022-11-23T01:05:34.9654238Z test_fsdp_memory_ckpt_ckpt (__main__.TestFSDPMemory) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1568 2022-11-23T01:05:34.9654626Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1569 2022-11-23T01:05:34.9655081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:34.9655413Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:34.9655835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:34.9656167Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:34.9656486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:05:34.9656939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:34.9657257Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:34.9657676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:34.9658026Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:34.9658365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:05:34.9658860Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:05:34.9659370Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:05:34.9659757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:05:34.9660104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:05:34.9660362Z dist init r=0, world=2 2022-11-23T01:05:34.9660526Z dist init r=1, world=2 2022-11-23T01:05:34.9660883Z skip: Need at least 2 CUDA devices (1.879s) 2022-11-23T01:05:34.9661214Z test_fsdp_memory_ckpt_no_ckpt (__main__.TestFSDPMemory) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1596 2022-11-23T01:05:34.9661575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1597 2022-11-23T01:05:34.9662027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:34.9662366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:34.9662786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:34.9663186Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:34.9663508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:05:34.9663963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:34.9664270Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:34.9664683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:34.9665012Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:34.9665330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:05:34.9665799Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:05:34.9666304Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:05:34.9666674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:05:34.9667012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:05:34.9667254Z dist init r=0, world=2 2022-11-23T01:05:34.9667425Z dist init r=1, world=2 2022-11-23T01:05:34.9667619Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:05:34.9667741Z 2022-11-23T01:05:34.9667931Z ---------------------------------------------------------------------- 2022-11-23T01:05:34.9668156Z Ran 2 tests in 3.493s 2022-11-23T01:05:34.9668262Z 2022-11-23T01:05:34.9668324Z OK (skipped=2) 2022-11-23T01:05:34.9668425Z 2022-11-23T01:05:34.9668497Z Generating XML reports... 2022-11-23T01:05:34.9668921Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_memory/TEST-TestFSDPMemory-20221123010531.xml 2022-11-23T01:05:34.9669170Z 2022-11-23T01:05:34.9669435Z ##[endgroup] 2022-11-23T01:05:34.9669865Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_memory (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_memory_o90wlh7b) 2022-11-23T01:05:34.9670120Z 2022-11-23T01:05:34.9670322Z Running distributed/checkpoint/test_checkpoint ... [2022-11-23 01:05:34.961572] 2022-11-23T01:05:34.9670844Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:05:34.961921] 2022-11-23T01:05:40.3040989Z 2022-11-23T01:05:40.3041582Z Expand the folded group to see the log file of distributed/checkpoint/test_checkpoint 2022-11-23T01:05:40.3042456Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_checkpoint_hwdwugiu) 2022-11-23T01:05:40.3043164Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_checkpoint 2022-11-23T01:05:40.3043384Z 2022-11-23T01:05:40.3043459Z Running tests... 2022-11-23T01:05:40.3043754Z ---------------------------------------------------------------------- 2022-11-23T01:05:40.3044334Z test_default_metadata (__main__.TestDistributedCheckpointing) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:05:40.3044791Z test_tensor_metadata_with_missing_rank_spec (__main__.TestDistributedCheckpointing) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:05:40.3045224Z test_dummy_reader_works (__main__.TestDistributedFailure) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-11-23T01:05:40.3045611Z test_dummy_writer_works (__main__.TestDistributedFailure) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-11-23T01:05:40.3046065Z test_load_error_handling (__main__.TestDistributedFailure) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:05:40.3046707Z test_load_error_handling_no_dist (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1635 2022-11-23T01:05:40.3047433Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1636 2022-11-23T01:05:40.3047759Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1637 2022-11-23T01:05:40.3048079Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1638 2022-11-23T01:05:40.3048557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:40.3048898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:40.3049458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:40.3049861Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:40.3050396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:05:40.3051090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:40.3051425Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:40.3051854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:40.3052202Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:40.3052512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:05:40.3052972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:40.3053304Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:40.3053717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:40.3054065Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:40.3054389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:05:40.3054852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:40.3055168Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:40.3055599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:40.3055946Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:40.3056267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:05:40.3056508Z ok (1.984s) 2022-11-23T01:05:40.3056801Z test_save_error_handling (__main__.TestDistributedFailure) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:05:40.3057248Z test_save_error_handling_no_dist (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1679 2022-11-23T01:05:40.3057690Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1680 2022-11-23T01:05:40.3058020Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1681 2022-11-23T01:05:40.3058339Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1682 2022-11-23T01:05:40.3058788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:40.3059105Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:40.3059534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:40.3059924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:40.3060239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:05:40.3060696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:40.3061024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:40.3061453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:40.3061782Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:40.3062105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:05:40.3062562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:40.3062889Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:40.3063310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:40.3063655Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:40.3063980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:05:40.3064425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:40.3064753Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:40.3065181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:40.3065525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:40.3065831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:05:40.3066089Z ok (1.720s) 2022-11-23T01:05:40.3066193Z 2022-11-23T01:05:40.3066397Z ---------------------------------------------------------------------- 2022-11-23T01:05:40.3066627Z Ran 8 tests in 3.710s 2022-11-23T01:05:40.3066741Z 2022-11-23T01:05:40.3066813Z OK (skipped=6) 2022-11-23T01:05:40.3066921Z 2022-11-23T01:05:40.3067006Z Generating XML reports... 2022-11-23T01:05:40.3067489Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_checkpoint/TEST-TestDistributedFailure-20221123010536.xml 2022-11-23T01:05:40.3068136Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_checkpoint/TEST-TestDistributedCheckpointing-20221123010536.xml 2022-11-23T01:05:40.3068433Z 2022-11-23T01:05:40.3068695Z ##[endgroup] 2022-11-23T01:05:40.3069174Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_checkpoint_hwdwugiu) 2022-11-23T01:05:40.3069452Z 2022-11-23T01:05:40.3069659Z Running distributed/fsdp/test_fsdp_clip_grad_norm ... [2022-11-23 01:05:40.304441] 2022-11-23T01:05:40.3070197Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_clip_grad_norm.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:05:40.304835] 2022-11-23T01:05:45.9322060Z 2022-11-23T01:05:45.9322857Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_clip_grad_norm 2022-11-23T01:05:45.9324058Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_clip_grad_norm (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_clip_grad_norm_f1ucc6eb) 2022-11-23T01:05:45.9325220Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm 2022-11-23T01:05:45.9325585Z 2022-11-23T01:05:45.9325662Z Running tests... 2022-11-23T01:05:45.9326001Z ---------------------------------------------------------------------- 2022-11-23T01:05:45.9326454Z test_ddp_parity (__main__.TestClipGradNorm) 2022-11-23T01:05:45.9327289Z Tests FSDP with ``FullyShardedDataParallel.clip_grad_norm_()` against ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1734 2022-11-23T01:05:45.9327768Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1735 2022-11-23T01:05:45.9328192Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1736 2022-11-23T01:05:45.9328751Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1737 2022-11-23T01:05:45.9329507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:45.9330112Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:45.9330775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:45.9331388Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:45.9331866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:05:45.9332343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:45.9332679Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:45.9333114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:45.9333446Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:45.9333781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:05:45.9334249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:45.9334581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:45.9334997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:45.9335350Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:45.9335688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:05:45.9336185Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:45.9336675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:45.9337011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:45.9337445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:45.9337781Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:45.9338120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:05:45.9338628Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:45.9339150Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:45.9339736Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:45.9340126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:05:45.9340485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:05:45.9340830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:05:45.9341163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:05:45.9341421Z dist init r=1, world=4 2022-11-23T01:05:45.9341640Z dist init r=2, world=4 2022-11-23T01:05:45.9341806Z dist init r=3, world=4 2022-11-23T01:05:45.9342020Z dist init r=0, world=4 2022-11-23T01:05:45.9342231Z skip: Need at least 2 CUDA devices (2.184s) 2022-11-23T01:05:45.9342474Z test_non_root (__main__.TestClipGradNorm) 2022-11-23T01:05:45.9342932Z Tests that calling ``clip_grad_norm_()`` on a non-root FSDP instance ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1790 2022-11-23T01:05:45.9343330Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1791 2022-11-23T01:05:45.9343661Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1792 2022-11-23T01:05:45.9343973Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1793 2022-11-23T01:05:45.9344429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:45.9344766Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:45.9345202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:45.9345539Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:45.9345880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:05:45.9346349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:45.9346678Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:45.9347090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:45.9347437Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:45.9347773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:05:45.9348233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:45.9348565Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:45.9348993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:45.9349340Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:45.9349659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:05:45.9350129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:45.9350457Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:45.9350868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:45.9351216Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:45.9351556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:05:45.9352173Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:45.9352686Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:45.9353203Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:45.9353722Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:45.9354109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:05:45.9354447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:05:45.9354824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:05:45.9355178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:05:45.9355426Z dist init r=3, world=4 2022-11-23T01:05:45.9355619Z dist init r=1, world=4 2022-11-23T01:05:45.9355802Z dist init r=0, world=4 2022-11-23T01:05:45.9355968Z dist init r=2, world=4 2022-11-23T01:05:45.9356175Z skip: Need at least 2 CUDA devices (1.820s) 2022-11-23T01:05:45.9356311Z 2022-11-23T01:05:45.9356514Z ---------------------------------------------------------------------- 2022-11-23T01:05:45.9356760Z Ran 2 tests in 4.004s 2022-11-23T01:05:45.9356861Z 2022-11-23T01:05:45.9356936Z OK (skipped=2) 2022-11-23T01:05:45.9357043Z 2022-11-23T01:05:45.9357131Z Generating XML reports... 2022-11-23T01:05:45.9357598Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestClipGradNorm-20221123010541.xml 2022-11-23T01:05:45.9357865Z 2022-11-23T01:05:45.9358140Z ##[endgroup] 2022-11-23T01:05:45.9358612Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_clip_grad_norm (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_clip_grad_norm_f1ucc6eb) 2022-11-23T01:05:45.9358889Z 2022-11-23T01:05:45.9361334Z Running distributed/_tensor/parallel/test_2d_parallel ... [2022-11-23 01:05:45.932578] 2022-11-23T01:05:45.9361894Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/parallel/test_2d_parallel.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:05:45.933011] 2022-11-23T01:05:51.5018632Z 2022-11-23T01:05:51.5019612Z Expand the folded group to see the log file of distributed/_tensor/parallel/test_2d_parallel 2022-11-23T01:05:51.5020830Z ##[group]PRINTING LOG FILE of distributed/_tensor/parallel/test_2d_parallel (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-parallel-test_2d_parallel_dp548oua) 2022-11-23T01:05:51.5021920Z Test results will be stored in test-reports/python-unittest/distributed._tensor.parallel.test_2d_parallel 2022-11-23T01:05:51.5022277Z 2022-11-23T01:05:51.5022391Z Running tests... 2022-11-23T01:05:51.5022900Z ---------------------------------------------------------------------- 2022-11-23T01:05:51.5023564Z test_2d_fsdp_integration_correctness (__main__.Test2dParallelIntegration) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1857 2022-11-23T01:05:51.5024226Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1858 2022-11-23T01:05:51.5024767Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1859 2022-11-23T01:05:51.5025273Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1860 2022-11-23T01:05:51.5025991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:51.5026522Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:51.5027273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:51.5027794Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:51.5028446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:05:51.5029098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:51.5029550Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:51.5030153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:51.5030642Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:51.5031118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:05:51.5032024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:51.5032500Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:51.5033154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:51.5033670Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:51.5034117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:05:51.5034761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:51.5035219Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:51.5035800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:51.5036272Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:51.5036726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:05:51.5037215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:05:51.5037706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:05:51.5038209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:05:51.5038718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:05:51.5039440Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:51.5040200Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:51.5041192Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:51.5041969Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:51.5042441Z skip: Need at least 4 CUDA devices (2.079s) 2022-11-23T01:05:51.5042983Z test_2d_fsdp_integration_functionality (__main__.Test2dParallelIntegration) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1913 2022-11-23T01:05:51.5043574Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1914 2022-11-23T01:05:51.5044057Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1915 2022-11-23T01:05:51.5044514Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1916 2022-11-23T01:05:51.5045190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:51.5045668Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:51.5046308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:51.5046785Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:51.5047373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:05:51.5048045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:51.5048520Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:51.5049151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:51.5049670Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:51.5050135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:05:51.5050878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:51.5051333Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:51.5051972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:51.5052460Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:51.5052891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:05:51.5053530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:51.5053965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:51.5054590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:51.5055083Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:51.5055531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:05:51.5056034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:05:51.5056574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:05:51.5057129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:05:51.5057957Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:51.5058586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:05:51.5059108Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:51.5059629Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:51.5060149Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:51.5060476Z skip: Need at least 4 CUDA devices (1.820s) 2022-11-23T01:05:51.5060616Z 2022-11-23T01:05:51.5060801Z ---------------------------------------------------------------------- 2022-11-23T01:05:51.5061048Z Ran 2 tests in 3.899s 2022-11-23T01:05:51.5061162Z 2022-11-23T01:05:51.5061235Z OK (skipped=2) 2022-11-23T01:05:51.5061342Z 2022-11-23T01:05:51.5061415Z Generating XML reports... 2022-11-23T01:05:51.5061911Z Generated XML report: test-reports/python-unittest/distributed._tensor.parallel.test_2d_parallel/TEST-Test2dParallelIntegration-20221123010547.xml 2022-11-23T01:05:51.5062206Z 2022-11-23T01:05:51.5062519Z ##[endgroup] 2022-11-23T01:05:51.5063016Z FINISHED PRINTING LOG FILE of distributed/_tensor/parallel/test_2d_parallel (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-parallel-test_2d_parallel_dp548oua) 2022-11-23T01:05:51.5063292Z 2022-11-23T01:05:51.5063503Z Running distributed/_composable/test_replicate ... [2022-11-23 01:05:51.502126] 2022-11-23T01:05:51.5064111Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_composable/test_replicate.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:05:51.502559] 2022-11-23T01:05:56.9249859Z 2022-11-23T01:05:56.9250493Z Expand the folded group to see the log file of distributed/_composable/test_replicate 2022-11-23T01:05:56.9251609Z ##[group]PRINTING LOG FILE of distributed/_composable/test_replicate (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_replicate_ul3_ntse) 2022-11-23T01:05:56.9252713Z Test results will be stored in test-reports/python-unittest/distributed._composable.test_replicate 2022-11-23T01:05:56.9252924Z 2022-11-23T01:05:56.9253246Z Running tests... 2022-11-23T01:05:56.9253560Z ---------------------------------------------------------------------- 2022-11-23T01:05:56.9254006Z test_replicate_multi_module (__main__.ReplicateTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1980 2022-11-23T01:05:56.9254489Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1981 2022-11-23T01:05:56.9255027Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1982 2022-11-23T01:05:56.9255917Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1983 2022-11-23T01:05:56.9256468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:56.9256806Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:56.9257298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:56.9257655Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:56.9258029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:05:56.9258528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:56.9258869Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:56.9259382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:56.9260041Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:56.9260635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:05:56.9261155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:56.9261474Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:56.9261965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:56.9262320Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:56.9262706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:05:56.9263199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:56.9263530Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:56.9264008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:56.9264345Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:56.9264714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:05:56.9265084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:05:56.9265503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:05:56.9265955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:05:56.9266512Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:56.9266954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:05:56.9267452Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:56.9268009Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:56.9268577Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:56.9269014Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration. 2022-11-23T01:05:56.9269399Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration. 2022-11-23T01:05:56.9269748Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration. 2022-11-23T01:05:56.9270141Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration. 2022-11-23T01:05:56.9270388Z ok (2.085s) 2022-11-23T01:05:56.9270725Z test_replicate_single_module (__main__.ReplicateTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2036 2022-11-23T01:05:56.9271126Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2037 2022-11-23T01:05:56.9271475Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2038 2022-11-23T01:05:56.9271802Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2039 2022-11-23T01:05:56.9272388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:56.9272722Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:56.9273204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:56.9273541Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:56.9273912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:05:56.9274373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:56.9274755Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:56.9275169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:56.9275570Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:56.9275923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:05:56.9276438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:56.9276776Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:56.9277228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:56.9277621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:56.9277929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:05:56.9278442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:05:56.9278777Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:05:56.9279254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:05:56.9279586Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:05:56.9280002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:05:56.9280366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:05:56.9280972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:05:56.9281323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:05:56.9281874Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:56.9282388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:05:56.9282874Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:56.9283445Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:56.9284004Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:05:56.9284391Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration. 2022-11-23T01:05:56.9284774Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration. 2022-11-23T01:05:56.9285122Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration. 2022-11-23T01:05:56.9285512Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration. 2022-11-23T01:05:56.9285760Z ok (1.719s) 2022-11-23T01:05:56.9285849Z 2022-11-23T01:05:56.9286063Z ---------------------------------------------------------------------- 2022-11-23T01:05:56.9286335Z Ran 2 tests in 3.804s 2022-11-23T01:05:56.9286449Z 2022-11-23T01:05:56.9286509Z OK 2022-11-23T01:05:56.9286597Z 2022-11-23T01:05:56.9286669Z Generating XML reports... 2022-11-23T01:05:56.9287170Z Generated XML report: test-reports/python-unittest/distributed._composable.test_replicate/TEST-ReplicateTest-20221123010552.xml 2022-11-23T01:05:56.9287425Z 2022-11-23T01:05:56.9287713Z ##[endgroup] 2022-11-23T01:05:56.9288219Z FINISHED PRINTING LOG FILE of distributed/_composable/test_replicate (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_replicate_ul3_ntse) 2022-11-23T01:05:56.9288494Z 2022-11-23T01:05:56.9288688Z Running distributed/_tensor/test_math_ops ... [2022-11-23 01:05:56.925427] 2022-11-23T01:05:56.9289244Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_math_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:05:56.925884] 2022-11-23T01:06:04.1792862Z 2022-11-23T01:06:04.1793342Z Expand the folded group to see the log file of distributed/_tensor/test_math_ops 2022-11-23T01:06:04.1794592Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_math_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_math_ops_0xdvq6xx) 2022-11-23T01:06:04.1795333Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_math_ops 2022-11-23T01:06:04.1795538Z 2022-11-23T01:06:04.1795615Z Running tests... 2022-11-23T01:06:04.1795923Z ---------------------------------------------------------------------- 2022-11-23T01:06:04.1796304Z test_softmax_fwd (__main__.DistMathOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2103 2022-11-23T01:06:04.1796669Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2104 2022-11-23T01:06:04.1797006Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2105 2022-11-23T01:06:04.1797336Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2106 2022-11-23T01:06:04.1798206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:04.1798587Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:04.1799030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:04.1799447Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:04.1799762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:06:04.1800289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:04.1800819Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:04.1801481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:04.1801895Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:04.1802225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:06:04.1802756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:04.1803076Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:04.1803565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:04.1803947Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:04.1804287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:06:04.1804797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:04.1805132Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:04.1805627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:04.1805976Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:04.1806344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:06:04.1806712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:06:04.1807143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:06:04.1807497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:06:04.1807917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:06:04.1808418Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:04.1809007Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:04.1809571Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:04.1810117Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:04.1810435Z ok (2.074s) 2022-11-23T01:06:04.1810744Z test_softmax_with_bwd (__main__.DistMathOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2159 2022-11-23T01:06:04.1811171Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2160 2022-11-23T01:06:04.1811502Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2161 2022-11-23T01:06:04.1811879Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2162 2022-11-23T01:06:04.1812318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:04.1812784Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:04.1813260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:04.1813627Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:04.1813938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:06:04.1814458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:04.1814828Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:04.1815306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:04.1815715Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:04.1816043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:06:04.1816566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:04.1816885Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:04.1817378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:04.1817726Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:04.1818096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:06:04.1818549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:04.1818940Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:04.1819377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:04.1819763Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:04.1820089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:06:04.1820520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:06:04.1820890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:06:04.1821301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:06:04.1821858Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:04.1822266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:06:04.1822826Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:04.1823395Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:04.1823919Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:04.1824302Z skip: Need at least 4 CUDA devices (1.820s) 2022-11-23T01:06:04.1824663Z test_sum (__main__.DistMathOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2215 2022-11-23T01:06:04.1825069Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2216 2022-11-23T01:06:04.1825403Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2217 2022-11-23T01:06:04.1825788Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2218 2022-11-23T01:06:04.1826272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:04.1826668Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:04.1827102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:04.1827511Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:04.1827823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:06:04.1828350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:04.1828685Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:04.1829203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:04.1829596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:04.1829927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:06:04.1830393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:04.1830775Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:04.1831206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:04.1831555Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:04.1831978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:06:04.1832443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:04.1832773Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:04.1833206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:04.1833552Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:04.1833859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:06:04.1834283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:06:04.1834655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:06:04.1835007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:06:04.1835506Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:04.1835909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:06:04.1836409Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:04.1836915Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:04.1837436Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:04.1837724Z ok (1.720s) 2022-11-23T01:06:04.1837837Z 2022-11-23T01:06:04.1838088Z ---------------------------------------------------------------------- 2022-11-23T01:06:04.1838322Z Ran 3 tests in 5.614s 2022-11-23T01:06:04.1838435Z 2022-11-23T01:06:04.1838509Z OK (skipped=1) 2022-11-23T01:06:04.1838616Z 2022-11-23T01:06:04.1838704Z Generating XML reports... 2022-11-23T01:06:04.1839136Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_math_ops/TEST-DistMathOpsTest-20221123010558.xml 2022-11-23T01:06:04.1839389Z 2022-11-23T01:06:04.1839676Z ##[endgroup] 2022-11-23T01:06:04.1840169Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_math_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_math_ops_0xdvq6xx) 2022-11-23T01:06:04.1840419Z 2022-11-23T01:06:04.1840721Z Running distributed/_tensor/test_pointwise_ops ... [2022-11-23 01:06:04.179661] 2022-11-23T01:06:04.1841255Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_pointwise_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:06:04.180039] 2022-11-23T01:06:13.5586068Z 2022-11-23T01:06:13.5591855Z Expand the folded group to see the log file of distributed/_tensor/test_pointwise_ops 2022-11-23T01:06:13.5593281Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_pointwise_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_pointwise_ops_19gxnylv) 2022-11-23T01:06:13.5594653Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_pointwise_ops 2022-11-23T01:06:13.5595042Z 2022-11-23T01:06:13.5595178Z Running tests... 2022-11-23T01:06:13.5595709Z ---------------------------------------------------------------------- 2022-11-23T01:06:13.5596425Z test_activations (__main__.DistElementwiseOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2282 2022-11-23T01:06:13.5597120Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2283 2022-11-23T01:06:13.5597687Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2284 2022-11-23T01:06:13.5598278Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2285 2022-11-23T01:06:13.5599129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:13.5599734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:13.5600513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:13.5601212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:13.5601662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:06:13.5602459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:13.5602890Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:13.5603321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:13.5603666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:13.5603978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:06:13.5604437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:13.5604762Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:13.5605194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:13.5605525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:13.5605847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:06:13.5606304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:13.5606618Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:13.5607044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:13.5607392Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:13.5607712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:06:13.5608170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:06:13.5608542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:06:13.5608904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:06:13.5609253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:06:13.5609748Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:13.5610383Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:13.5610955Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:13.5611456Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:13.5611742Z ok (2.273s) 2022-11-23T01:06:13.5612079Z test_dropout (__main__.DistElementwiseOpsTest) ... skip: testing RNG based ops is broken: https://github.com/pytorch/tau/issues/494 (0.001s) 2022-11-23T01:06:13.5612540Z test_dropout_backward (__main__.DistElementwiseOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2350 2022-11-23T01:06:13.5612914Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2351 2022-11-23T01:06:13.5613240Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2352 2022-11-23T01:06:13.5613564Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2353 2022-11-23T01:06:13.5614015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:13.5614332Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:13.5614762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:13.5615107Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:13.5615419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:06:13.5615875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:13.5616202Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:13.5616628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:13.5616961Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:13.5617283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:06:13.5617740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:13.5618056Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:13.5618478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:13.5618826Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:13.5619146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:06:13.5619592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:13.5619920Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:13.5620344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:13.5620687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:13.5621074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:06:13.5621490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:06:13.5621857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:06:13.5622206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:06:13.5622567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:06:13.5623058Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:13.5623616Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:13.5624122Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:13.5624639Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:13.5624962Z skip: Need at least 4 CUDA devices (1.819s) 2022-11-23T01:06:13.5625316Z test_dropout_errors (__main__.DistElementwiseOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2406 2022-11-23T01:06:13.5625691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2407 2022-11-23T01:06:13.5626017Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2408 2022-11-23T01:06:13.5626342Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2409 2022-11-23T01:06:13.5626778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:13.5627104Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:13.5627532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:13.5627879Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:13.5628189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:06:13.5628646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:13.5628973Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:13.5629388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:13.5629738Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:13.5630056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:06:13.5630511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:13.5630824Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:13.5631250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:13.5631591Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:13.5631983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:06:13.5632431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:13.5632761Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:13.5633187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:13.5633515Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:13.5633875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:06:13.5634239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:06:13.5634606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:06:13.5634958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:06:13.5635449Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:13.5635875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:06:13.5636366Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:13.5636868Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:13.5637382Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:13.5637668Z ok (1.820s) 2022-11-23T01:06:13.5637966Z test_mul_out (__main__.DistElementwiseOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2462 2022-11-23T01:06:13.5638347Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2463 2022-11-23T01:06:13.5638672Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2464 2022-11-23T01:06:13.5638990Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2465 2022-11-23T01:06:13.5639426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:13.5639759Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:13.5640191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:13.5640523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:13.5640995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:06:13.5641457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:13.5641787Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:13.5642200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:13.5642548Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:13.5642872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:06:13.5643335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:13.5643650Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:13.5644076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:13.5644419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:13.5644728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:06:13.5645185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:13.5645515Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:13.5645939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:13.5646267Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:13.5646672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:06:13.5647039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:06:13.5647394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:06:13.5647756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:06:13.5648112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:06:13.5648604Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:13.5649150Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:13.5649667Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:13.5650179Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:13.5650467Z ok (1.820s) 2022-11-23T01:06:13.5650559Z 2022-11-23T01:06:13.5650759Z ---------------------------------------------------------------------- 2022-11-23T01:06:13.5650999Z Ran 5 tests in 7.734s 2022-11-23T01:06:13.5651113Z 2022-11-23T01:06:13.5651185Z OK (skipped=2) 2022-11-23T01:06:13.5651292Z 2022-11-23T01:06:13.5651362Z Generating XML reports... 2022-11-23T01:06:13.5651911Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_pointwise_ops/TEST-DistElementwiseOpsTest-20221123010605.xml 2022-11-23T01:06:13.5652185Z 2022-11-23T01:06:13.5652484Z ##[endgroup] 2022-11-23T01:06:13.5652935Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_pointwise_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_pointwise_ops_19gxnylv) 2022-11-23T01:06:13.5653197Z 2022-11-23T01:06:13.5653438Z Running distributed/_tensor/parallel/test_parallelize_api ... [2022-11-23 01:06:13.559248] 2022-11-23T01:06:13.5653995Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/parallel/test_parallelize_api.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:06:13.559610] 2022-11-23T01:06:22.9473339Z 2022-11-23T01:06:22.9473910Z Expand the folded group to see the log file of distributed/_tensor/parallel/test_parallelize_api 2022-11-23T01:06:22.9474976Z ##[group]PRINTING LOG FILE of distributed/_tensor/parallel/test_parallelize_api (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-parallel-test_parallelize_api_o7ebzpgo) 2022-11-23T01:06:22.9475996Z Test results will be stored in test-reports/python-unittest/distributed._tensor.parallel.test_parallelize_api 2022-11-23T01:06:22.9476308Z 2022-11-23T01:06:22.9476410Z Running tests... 2022-11-23T01:06:22.9476929Z ---------------------------------------------------------------------- 2022-11-23T01:06:22.9477586Z test_creat_1d_device_mesh (__main__.TensorParallelAPITests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2529 2022-11-23T01:06:22.9478224Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2530 2022-11-23T01:06:22.9478751Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2531 2022-11-23T01:06:22.9479284Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2532 2022-11-23T01:06:22.9480118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:22.9480914Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:22.9481708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:22.9483436Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:22.9484262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:06:22.9489506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:22.9490147Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:22.9491167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:22.9492772Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:22.9493291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:06:22.9494397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:22.9494989Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:22.9495755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:22.9496357Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:22.9496923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:06:22.9497877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:22.9498480Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:22.9499187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:22.9499766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:22.9500346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:06:22.9500998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:06:22.9501648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:06:22.9502339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:06:22.9502967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:06:22.9503888Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:22.9504702Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:22.9505531Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:22.9506350Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:22.9506948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2022-11-23T01:06:22.9507514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T01:06:22.9508108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T01:06:22.9508692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-11-23T01:06:22.9509479Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T01:06:22.9510311Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T01:06:22.9510909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T01:06:22.9511609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2 2022-11-23T01:06:22.9512537Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T01:06:22.9513393Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T01:06:22.9514035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T01:06:22.9514698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3 2022-11-23T01:06:22.9515592Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes. 2022-11-23T01:06:22.9516655Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes. 2022-11-23T01:06:22.9517382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3 2022-11-23T01:06:22.9518252Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes. 2022-11-23T01:06:22.9519176Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes. 2022-11-23T01:06:22.9519869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2 2022-11-23T01:06:22.9520452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T01:06:22.9521189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T01:06:22.9522039Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes. 2022-11-23T01:06:22.9522858Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes. 2022-11-23T01:06:22.9523507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0 2022-11-23T01:06:22.9524275Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes. 2022-11-23T01:06:22.9525058Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes. 2022-11-23T01:06:22.9525671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1 2022-11-23T01:06:22.9526195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2 2022-11-23T01:06:22.9526771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3 2022-11-23T01:06:22.9527519Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes. 2022-11-23T01:06:22.9528302Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes. 2022-11-23T01:06:22.9529111Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes. 2022-11-23T01:06:22.9529873Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes. 2022-11-23T01:06:22.9530350Z ok (2.274s) 2022-11-23T01:06:22.9530869Z test_creat_1d_device_mesh_error (__main__.TensorParallelAPITests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2609 2022-11-23T01:06:22.9531461Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2610 2022-11-23T01:06:22.9531954Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2611 2022-11-23T01:06:22.9532424Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2612 2022-11-23T01:06:22.9533130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:22.9533460Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:22.9533893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:22.9534243Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:22.9534571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:06:22.9535023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:22.9535399Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:22.9535826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:22.9536157Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:22.9536482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:06:22.9536941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:22.9537269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:22.9537681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:22.9538020Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:22.9538340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:06:22.9538787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:22.9539111Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:22.9539537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:22.9539879Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:22.9540186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:06:22.9540549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:06:22.9540916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:06:22.9541273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:06:22.9541751Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:22.9542145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:06:22.9542633Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:22.9543147Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:22.9543648Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:22.9543933Z ok (1.819s) 2022-11-23T01:06:22.9544256Z test_parallelize_mlp (__main__.TensorParallelAPITests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2665 2022-11-23T01:06:22.9544637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2666 2022-11-23T01:06:22.9544965Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2667 2022-11-23T01:06:22.9545285Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2668 2022-11-23T01:06:22.9545769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:22.9546087Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:22.9546515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:22.9546858Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:22.9547168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:06:22.9547626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:22.9547983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:22.9548409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:22.9548742Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:22.9549061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:06:22.9549517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:22.9549842Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:22.9550255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:22.9550597Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:22.9550914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:06:22.9551356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:22.9551678Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:22.9552196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:22.9552540Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:22.9552848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:06:22.9553211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:06:22.9553578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:06:22.9553929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:06:22.9554423Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:22.9554821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:06:22.9555316Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:22.9555822Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:22.9556335Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:22.9556622Z ok (1.820s) 2022-11-23T01:06:22.9556951Z test_parallelize_mlp_error (__main__.TensorParallelAPITests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2721 2022-11-23T01:06:22.9557336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2722 2022-11-23T01:06:22.9557665Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2723 2022-11-23T01:06:22.9557981Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2724 2022-11-23T01:06:22.9558461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:22.9558788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:22.9559217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:22.9559562Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:22.9559871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:06:22.9560329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:22.9560860Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:22.9561300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:22.9561633Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:22.9561954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:06:22.9562408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:22.9562723Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:22.9563148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:22.9563491Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:22.9563812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:06:22.9564257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:22.9564579Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:22.9565012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:22.9565344Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:22.9565661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:06:22.9566021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:06:22.9566388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:06:22.9566737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:06:22.9567096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:06:22.9567585Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:22.9568104Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:22.9568606Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:22.9569113Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:22.9569398Z ok (1.820s) 2022-11-23T01:06:22.9569501Z 2022-11-23T01:06:22.9569688Z ---------------------------------------------------------------------- 2022-11-23T01:06:22.9569931Z Ran 4 tests in 7.733s 2022-11-23T01:06:22.9570044Z 2022-11-23T01:06:22.9570104Z OK 2022-11-23T01:06:22.9570196Z 2022-11-23T01:06:22.9570281Z Generating XML reports... 2022-11-23T01:06:22.9570766Z Generated XML report: test-reports/python-unittest/distributed._tensor.parallel.test_parallelize_api/TEST-TensorParallelAPITests-20221123010614.xml 2022-11-23T01:06:22.9571054Z 2022-11-23T01:06:22.9571381Z ##[endgroup] 2022-11-23T01:06:22.9571987Z FINISHED PRINTING LOG FILE of distributed/_tensor/parallel/test_parallelize_api (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-parallel-test_parallelize_api_o7ebzpgo) 2022-11-23T01:06:22.9572283Z 2022-11-23T01:06:22.9572507Z Running distributed/checkpoint/test_file_system_checkpoint_cpu ... [2022-11-23 01:06:22.947718] 2022-11-23T01:06:22.9573077Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint_cpu.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:06:22.948114] 2022-11-23T01:06:33.2477715Z 2022-11-23T01:06:33.2478589Z Expand the folded group to see the log file of distributed/checkpoint/test_file_system_checkpoint_cpu 2022-11-23T01:06:33.2480115Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_file_system_checkpoint_cpu (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_file_system_checkpoint_cpu_00vso8cs) 2022-11-23T01:06:33.2481474Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu 2022-11-23T01:06:33.2481903Z 2022-11-23T01:06:33.2482003Z Running tests... 2022-11-23T01:06:33.2482502Z ---------------------------------------------------------------------- 2022-11-23T01:06:33.2483193Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2788 2022-11-23T01:06:33.2483845Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2789 2022-11-23T01:06:33.2484602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:33.2485175Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:33.2485848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:33.2486383Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:33.2486897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:06:33.2487668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:33.2488163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:33.2488729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:33.2489193Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:33.2489629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:06:33.2490126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:06:33.2490628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:06:33.2491381Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:06:33.2492208Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:06:33.2492642Z ok (1.886s) 2022-11-23T01:06:33.2493156Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2816 2022-11-23T01:06:33.2493786Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2817 2022-11-23T01:06:33.2494449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:33.2494900Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:33.2495498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:33.2496140Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:33.2496576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:06:33.2497226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:33.2497669Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:33.2498270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:33.2498719Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:33.2499258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:06:33.2499787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:06:33.2500316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:06:33.2501038Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:06:33.2501824Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:06:33.2502245Z ok (1.715s) 2022-11-23T01:06:33.2502694Z test_save_load_bytes (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2844 2022-11-23T01:06:33.2503261Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2845 2022-11-23T01:06:33.2503940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:33.2504419Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:33.2505032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:33.2505527Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:33.2505998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:06:33.2506655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:33.2507145Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:33.2507777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:33.2508281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:33.2508746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:06:33.2509289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:06:33.2509814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:06:33.2510537Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:06:33.2511269Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:06:33.2511691Z ok (1.613s) 2022-11-23T01:06:33.2512281Z test_switch_between_sharded_tensor_to_tensor (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2872 2022-11-23T01:06:33.2512849Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2873 2022-11-23T01:06:33.2513480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:33.2513935Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:33.2514638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:33.2515115Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:33.2515562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:06:33.2516216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:33.2516700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:33.2517386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:33.2518035Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:33.2518466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:06:33.2518817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:06:33.2519181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:06:33.2519682Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:06:33.2520201Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:06:33.2520477Z ok (1.814s) 2022-11-23T01:06:33.2520886Z test_read_write_only_tensor (__main__.TestDistributedStateDictSaveLoad) ... ok (0.017s) 2022-11-23T01:06:33.2521378Z test_read_write_shard_tensor (__main__.TestDistributedStateDictSaveLoadWithSharedTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2900 2022-11-23T01:06:33.2521839Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2901 2022-11-23T01:06:33.2522285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:33.2522611Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:33.2523034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:33.2523366Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:33.2523689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:06:33.2524144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:33.2524468Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:33.2524883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:33.2525225Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:33.2525549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:06:33.2525898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:06:33.2526259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:06:33.2526751Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:06:33.2527267Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:06:33.2527540Z ok (1.613s) 2022-11-23T01:06:33.2527643Z 2022-11-23T01:06:33.2527843Z ---------------------------------------------------------------------- 2022-11-23T01:06:33.2528082Z Ran 6 tests in 8.658s 2022-11-23T01:06:33.2528197Z 2022-11-23T01:06:33.2528255Z OK 2022-11-23T01:06:33.2528332Z 2022-11-23T01:06:33.2528504Z Generating XML reports... 2022-11-23T01:06:33.2529041Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedReshardOnLoad-20221123010624.xml 2022-11-23T01:06:33.2529761Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoad-20221123010624.xml 2022-11-23T01:06:33.2530544Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20221123010624.xml 2022-11-23T01:06:33.2530918Z 2022-11-23T01:06:33.2531266Z ##[endgroup] 2022-11-23T01:06:33.2531790Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_file_system_checkpoint_cpu (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_file_system_checkpoint_cpu_00vso8cs) 2022-11-23T01:06:33.2532095Z 2022-11-23T01:06:33.2532297Z Running distributed/_tensor/test_api ... [2022-11-23 01:06:33.248157] 2022-11-23T01:06:33.2532784Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_api.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:06:33.248589] 2022-11-23T01:06:44.7620829Z 2022-11-23T01:06:44.7621399Z Expand the folded group to see the log file of distributed/_tensor/test_api 2022-11-23T01:06:44.7622566Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_api (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_api_zx7pfywc) 2022-11-23T01:06:44.7623626Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_api 2022-11-23T01:06:44.7623982Z 2022-11-23T01:06:44.7624129Z Running tests... 2022-11-23T01:06:44.7624683Z ---------------------------------------------------------------------- 2022-11-23T01:06:44.7625344Z test_distribute_module (__main__.DTensorAPITest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2939 2022-11-23T01:06:44.7625954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2940 2022-11-23T01:06:44.7626503Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2941 2022-11-23T01:06:44.7627063Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2942 2022-11-23T01:06:44.7627830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:44.7628349Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:44.7629090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:44.7629616Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:44.7630117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:06:44.7630892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:44.7631432Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:44.7632168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:44.7632612Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:44.7633047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:06:44.7633689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:44.7634142Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:44.7634752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:44.7635260Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:44.7635997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:06:44.7636713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:44.7637192Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:44.7637850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:44.7638329Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:44.7638762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:06:44.7639361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:06:44.7639860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:06:44.7640498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:06:44.7641414Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:44.7641955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:06:44.7642623Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:44.7643360Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:44.7644118Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:44.7644501Z ok (2.282s) 2022-11-23T01:06:44.7644984Z test_distribute_module_input_fn_output_fn (__main__.DTensorAPITest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2995 2022-11-23T01:06:44.7645566Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2996 2022-11-23T01:06:44.7646047Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2997 2022-11-23T01:06:44.7646499Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2998 2022-11-23T01:06:44.7647156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:44.7647645Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:44.7648277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:44.7648761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:44.7649228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:06:44.7649893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:44.7650354Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:44.7650983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:44.7651480Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:44.7651932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:06:44.7652585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:44.7653084Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:44.7653713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:44.7654198Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:44.7654765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:06:44.7655440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:44.7655906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:44.7656486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:44.7656949Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:44.7657376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:06:44.7657949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:06:44.7658457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:06:44.7658975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:06:44.7659477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:06:44.7660147Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:44.7660932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:44.7661788Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:44.7662494Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:44.7662774Z ok (1.920s) 2022-11-23T01:06:44.7663086Z test_distribute_tensor (__main__.DTensorAPITest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3051 2022-11-23T01:06:44.7663473Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3052 2022-11-23T01:06:44.7663804Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3053 2022-11-23T01:06:44.7664115Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3054 2022-11-23T01:06:44.7664563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:44.7664890Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:44.7665306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:44.7665652Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:44.7665971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:06:44.7666423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:44.7666736Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:44.7667164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:44.7667507Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:44.7667812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:06:44.7668266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:44.7668588Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:44.7669010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:44.7669338Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:44.7669718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:06:44.7670174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:44.7670500Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:44.7670909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:44.7671253Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:44.7671573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:06:44.7672068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:06:44.7672435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:06:44.7672800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:06:44.7673159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:06:44.7673645Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:44.7674162Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:44.7674675Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:44.7675185Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:44.7675455Z ok (1.820s) 2022-11-23T01:06:44.7675768Z test_distribute_tensor_errors (__main__.DTensorAPITest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3107 2022-11-23T01:06:44.7676154Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3108 2022-11-23T01:06:44.7676464Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3109 2022-11-23T01:06:44.7676787Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3110 2022-11-23T01:06:44.7677233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:44.7677558Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:44.7677972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:44.7678317Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:44.7678636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:06:44.7679075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:44.7679403Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:44.7679826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:44.7680167Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:44.7680472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:06:44.7681074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:44.7681403Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:44.7681830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:44.7682159Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:44.7682549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:06:44.7683007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:44.7683321Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:44.7683742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:44.7684086Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:44.7684405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:06:44.7684796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:06:44.7685251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:06:44.7685848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:06:44.7686200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:06:44.7686694Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:44.7687209Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:44.7687722Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:44.7688219Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:44.7688610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2022-11-23T01:06:44.7688972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T01:06:44.7689330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-11-23T01:06:44.7689672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T01:06:44.7690156Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T01:06:44.7690668Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T01:06:44.7691056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T01:06:44.7691530Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T01:06:44.7692032Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T01:06:44.7692420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2 2022-11-23T01:06:44.7692775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3 2022-11-23T01:06:44.7693121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T01:06:44.7693606Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes. 2022-11-23T01:06:44.7694111Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes. 2022-11-23T01:06:44.7694619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes. 2022-11-23T01:06:44.7695112Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes. 2022-11-23T01:06:44.7695546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2 2022-11-23T01:06:44.7695904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3 2022-11-23T01:06:44.7696249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T01:06:44.7696603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T01:06:44.7697083Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes. 2022-11-23T01:06:44.7697589Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes. 2022-11-23T01:06:44.7698000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0 2022-11-23T01:06:44.7698359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1 2022-11-23T01:06:44.7698843Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes. 2022-11-23T01:06:44.7699354Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes. 2022-11-23T01:06:44.7699731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2 2022-11-23T01:06:44.7700089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3 2022-11-23T01:06:44.7700569Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes. 2022-11-23T01:06:44.7701080Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes. 2022-11-23T01:06:44.7701573Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes. 2022-11-23T01:06:44.7702082Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes. 2022-11-23T01:06:44.7702364Z ok (1.920s) 2022-11-23T01:06:44.7702677Z test_distribute_tensor_uneven_sharding (__main__.DTensorAPITest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3187 2022-11-23T01:06:44.7703070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3188 2022-11-23T01:06:44.7703395Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3189 2022-11-23T01:06:44.7703713Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3190 2022-11-23T01:06:44.7704148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:44.7704477Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:44.7704911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:44.7705258Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:44.7705570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:06:44.7706033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:44.7706367Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:44.7706784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:44.7707134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:44.7707456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:06:44.7707916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:44.7708262Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:44.7708691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:44.7709037Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:44.7709343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:06:44.7709802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:44.7710130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:44.7710581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:44.7710913Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:44.7711237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:06:44.7711602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:06:44.7712066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:06:44.7712420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:06:44.7712917Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:44.7713318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:06:44.7713797Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:44.7714315Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:44.7714836Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:44.7715124Z ok (1.920s) 2022-11-23T01:06:44.7715232Z 2022-11-23T01:06:44.7715421Z ---------------------------------------------------------------------- 2022-11-23T01:06:44.7715661Z Ran 5 tests in 9.863s 2022-11-23T01:06:44.7715775Z 2022-11-23T01:06:44.7715834Z OK 2022-11-23T01:06:44.7715923Z 2022-11-23T01:06:44.7715994Z Generating XML reports... 2022-11-23T01:06:44.7716417Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_api/TEST-DTensorAPITest-20221123010634.xml 2022-11-23T01:06:44.7716660Z 2022-11-23T01:06:44.7716986Z ##[endgroup] 2022-11-23T01:06:44.7717403Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_api (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_api_zx7pfywc) 2022-11-23T01:06:44.7717646Z 2022-11-23T01:06:44.7717855Z Running distributed/_tensor/test_tp_sharding_ops ... [2022-11-23 01:06:44.762569] 2022-11-23T01:06:44.7718392Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_tp_sharding_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:06:44.762979] 2022-11-23T01:06:57.6850798Z 2022-11-23T01:06:57.6851658Z Expand the folded group to see the log file of distributed/_tensor/test_tp_sharding_ops 2022-11-23T01:06:57.6852790Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_tp_sharding_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_tp_sharding_ops_2sx3lvxd) 2022-11-23T01:06:57.6853905Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_tp_sharding_ops 2022-11-23T01:06:57.6854381Z 2022-11-23T01:06:57.6854458Z Running tests... 2022-11-23T01:06:57.6854773Z ---------------------------------------------------------------------- 2022-11-23T01:06:57.6855366Z test_replicated_permute (__main__.TPShardingOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3254 2022-11-23T01:06:57.6855761Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3255 2022-11-23T01:06:57.6856092Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3256 2022-11-23T01:06:57.6856417Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3257 2022-11-23T01:06:57.6856872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:57.6857205Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:57.6857640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:57.6858084Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:57.6858562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:06:57.6859287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:57.6859688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:57.6860108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:57.6860456Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:57.6860781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:06:57.6861228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:57.6861668Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:57.6862210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:57.6862556Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:57.6862867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:06:57.6863320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:57.6863645Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:57.6864070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:57.6864399Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:57.6864719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:06:57.6865083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:06:57.6865435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:06:57.6865795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:06:57.6866290Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:57.6866685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:06:57.6867161Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:57.6867669Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:57.6868184Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:57.6868465Z ok (2.074s) 2022-11-23T01:06:57.6868829Z test_sharded_cat (__main__.TPShardingOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3310 2022-11-23T01:06:57.6869212Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3311 2022-11-23T01:06:57.6879228Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3312 2022-11-23T01:06:57.6879699Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3313 2022-11-23T01:06:57.6880214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:57.6880554Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:57.6881173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:57.6881650Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:57.6881978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:06:57.6882450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:57.6882770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:57.6883199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:57.6883545Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:57.6883866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:06:57.6884312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:57.6884639Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:57.6885063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:57.6885392Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:57.6885714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:06:57.6886171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:57.6886492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:57.6886904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:57.6887242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:57.6887563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:06:57.6887918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:06:57.6888286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:06:57.6888654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:06:57.6889146Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:57.6889655Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:57.6890049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:06:57.6890538Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:57.6891048Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:57.6891318Z ok (1.820s) 2022-11-23T01:06:57.6891631Z test_sharded_permute (__main__.TPShardingOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3366 2022-11-23T01:06:57.6892068Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3367 2022-11-23T01:06:57.6892399Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3368 2022-11-23T01:06:57.6892711Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3369 2022-11-23T01:06:57.6893156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:57.6893487Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:57.6893902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:57.6894287Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:57.6894611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:06:57.6895070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:57.6895424Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:57.6896017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:57.6896359Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:57.6896665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:06:57.6897120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:57.6897446Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:57.6897872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:57.6898201Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:57.6898522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:06:57.6898978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:57.6899292Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:57.6899716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:57.6900057Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:57.6900373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:06:57.6900724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:06:57.6901088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:06:57.6901453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:06:57.6901814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:06:57.6902287Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:57.6902801Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:57.6903311Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:57.6903814Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:57.6904085Z ok (1.819s) 2022-11-23T01:06:57.6904391Z test_sharded_split (__main__.TPShardingOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3422 2022-11-23T01:06:57.6904817Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3423 2022-11-23T01:06:57.6905134Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3424 2022-11-23T01:06:57.6905453Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3425 2022-11-23T01:06:57.6905902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:57.6906232Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:57.6906644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:57.6907020Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:57.6907340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:06:57.6907784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:57.6908111Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:57.6908534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:57.6908875Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:57.6909181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:06:57.6909636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:57.6909966Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:57.6910378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:57.6910717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:57.6911040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:06:57.6911491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:57.6911806Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:57.6912306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:57.6912650Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:57.6912971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:06:57.6913322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:06:57.6913696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:06:57.6914069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:06:57.6914551Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:57.6915069Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:57.6915465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:06:57.6915948Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:57.6916445Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:57.6916732Z ok (1.820s) 2022-11-23T01:06:57.6917047Z test_sharded_transpose (__main__.TPShardingOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3478 2022-11-23T01:06:57.6917471Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3479 2022-11-23T01:06:57.6917788Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3480 2022-11-23T01:06:57.6918108Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3481 2022-11-23T01:06:57.6918555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:57.6918868Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:57.6919292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:57.6919671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:57.6919996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:06:57.6920444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:57.6920910Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:57.6921342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:57.6921691Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:57.6922002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:06:57.6922461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:57.6922789Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:57.6923203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:57.6923549Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:57.6923875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:06:57.6924325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:57.6924635Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:57.6925055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:57.6925396Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:57.6925703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:06:57.6926062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:06:57.6926426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:06:57.6926796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:06:57.6927145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:06:57.6927630Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:57.6928149Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:57.6928661Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:57.6929157Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:57.6929446Z ok (1.920s) 2022-11-23T01:06:57.6929755Z test_sharded_view (__main__.TPShardingOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3534 2022-11-23T01:06:57.6930182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3535 2022-11-23T01:06:57.6930512Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3536 2022-11-23T01:06:57.6930834Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3537 2022-11-23T01:06:57.6931281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:57.6931594Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:57.6932020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:57.6932413Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:57.6932722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:06:57.6933180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:57.6933504Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:57.6933930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:57.6934260Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:57.6934582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:06:57.6935035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:57.6935359Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:57.6935773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:57.6936114Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:57.6936438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:06:57.6936877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:06:57.6937200Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:06:57.6937619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:06:57.6937958Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:06:57.6938269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:06:57.6938629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:06:57.6938994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:06:57.6939366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:06:57.6939714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:06:57.6940205Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:57.6940721Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:57.6941219Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:57.6941729Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:06:57.6942016Z ok (1.821s) 2022-11-23T01:06:57.6942119Z 2022-11-23T01:06:57.6942315Z ---------------------------------------------------------------------- 2022-11-23T01:06:57.6942544Z Ran 6 tests in 11.274s 2022-11-23T01:06:57.6942661Z 2022-11-23T01:06:57.6942777Z OK 2022-11-23T01:06:57.6942869Z 2022-11-23T01:06:57.6942955Z Generating XML reports... 2022-11-23T01:06:57.6943399Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_tp_sharding_ops/TEST-TPShardingOpsTest-20221123010646.xml 2022-11-23T01:06:57.6943660Z 2022-11-23T01:06:57.6943990Z ##[endgroup] 2022-11-23T01:06:57.6944463Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_tp_sharding_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_tp_sharding_ops_2sx3lvxd) 2022-11-23T01:06:57.6944727Z 2022-11-23T01:06:57.6944940Z Running distributed/fsdp/test_fsdp_meta ... [2022-11-23 01:06:57.685728] 2022-11-23T01:06:57.6945470Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_meta.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:06:57.686075] 2022-11-23T01:07:10.9618512Z 2022-11-23T01:07:10.9619208Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_meta 2022-11-23T01:07:10.9620299Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_meta (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_meta_gbzoc299) 2022-11-23T01:07:10.9626151Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_meta 2022-11-23T01:07:10.9626541Z 2022-11-23T01:07:10.9626680Z Running tests... 2022-11-23T01:07:10.9627329Z ---------------------------------------------------------------------- 2022-11-23T01:07:10.9628075Z test_bad_arg_meta (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3601 2022-11-23T01:07:10.9628753Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3602 2022-11-23T01:07:10.9629549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:10.9630105Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:10.9630915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:10.9631518Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:10.9632200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:07:10.9633070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:10.9633658Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:10.9634431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:10.9635059Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:10.9635599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:07:10.9636488Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:07:10.9637440Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:07:10.9638176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:07:10.9638838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:07:10.9639278Z dist init r=1, world=2 2022-11-23T01:07:10.9639588Z dist init r=0, world=2 2022-11-23T01:07:10.9639952Z skip: Need at least 2 CUDA devices (1.975s) 2022-11-23T01:07:10.9640829Z test_bad_arg_torchdistx (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s) 2022-11-23T01:07:10.9641822Z test_nested_model_with_meta_device_default_init_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3629 2022-11-23T01:07:10.9642512Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3630 2022-11-23T01:07:10.9643268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:10.9643791Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:10.9644482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:10.9645043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:10.9645575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:07:10.9646474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:10.9646983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:10.9647648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:10.9648228Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:10.9648777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:07:10.9649608Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:07:10.9650474Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:07:10.9651153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:07:10.9651805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:07:10.9652265Z dist init r=1, world=2 2022-11-23T01:07:10.9652573Z dist init r=0, world=2 2022-11-23T01:07:10.9652919Z skip: Need at least 2 CUDA devices (1.612s) 2022-11-23T01:07:10.9653638Z test_nested_model_with_meta_device_default_init_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3657 2022-11-23T01:07:10.9654387Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3658 2022-11-23T01:07:10.9655203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:10.9655801Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:10.9656551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:10.9657102Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:10.9657646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:07:10.9658449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:10.9658982Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:10.9659674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:10.9660227Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:10.9660739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:07:10.9661510Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:07:10.9662334Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:07:10.9662910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:07:10.9663563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:07:10.9663969Z dist init r=1, world=2 2022-11-23T01:07:10.9664221Z dist init r=0, world=2 2022-11-23T01:07:10.9664526Z skip: Need at least 2 CUDA devices (1.613s) 2022-11-23T01:07:10.9665171Z test_nested_model_with_meta_device_reset_params_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3685 2022-11-23T01:07:10.9665839Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3686 2022-11-23T01:07:10.9666562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:10.9667162Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:10.9667821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:10.9668331Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:10.9668818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:07:10.9669352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:10.9669691Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:10.9670114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:10.9670469Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:10.9670813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:07:10.9671300Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:07:10.9671830Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:07:10.9672304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:07:10.9672660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:07:10.9672910Z dist init r=0, world=2 2022-11-23T01:07:10.9673096Z dist init r=1, world=2 2022-11-23T01:07:10.9673313Z skip: Need at least 2 CUDA devices (1.612s) 2022-11-23T01:07:10.9673701Z test_nested_model_with_meta_device_reset_params_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3713 2022-11-23T01:07:10.9674142Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3714 2022-11-23T01:07:10.9674602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:10.9674941Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:10.9675362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:10.9675714Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:10.9676056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:07:10.9676515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:10.9676849Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:10.9677281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:10.9677631Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:10.9677955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:07:10.9678520Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:07:10.9679049Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:07:10.9679486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:07:10.9679825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:07:10.9680090Z dist init r=1, world=2 2022-11-23T01:07:10.9680275Z dist init r=0, world=2 2022-11-23T01:07:10.9680487Z skip: Need at least 2 CUDA devices (1.612s) 2022-11-23T01:07:10.9681073Z test_nested_model_with_torchdistX_default_init_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s) 2022-11-23T01:07:10.9681629Z test_nested_model_with_torchdistX_default_init_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s) 2022-11-23T01:07:10.9682169Z test_nested_model_with_torchdistX_init_fn_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s) 2022-11-23T01:07:10.9682700Z test_nested_model_with_torchdistX_init_fn_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s) 2022-11-23T01:07:10.9683214Z test_simple_model_with_meta_device_default_init (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3741 2022-11-23T01:07:10.9683638Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3742 2022-11-23T01:07:10.9684108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:10.9684449Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:10.9684871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:10.9685224Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:10.9685564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:07:10.9686023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:10.9686356Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:10.9686786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:10.9687133Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:10.9687456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:07:10.9687957Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:07:10.9688480Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:07:10.9688869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:07:10.9689207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:07:10.9689463Z dist init r=1, world=2 2022-11-23T01:07:10.9689651Z dist init r=0, world=2 2022-11-23T01:07:10.9689855Z skip: Need at least 2 CUDA devices (1.612s) 2022-11-23T01:07:10.9690247Z test_simple_model_with_meta_device_reset_params (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3769 2022-11-23T01:07:10.9690732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3770 2022-11-23T01:07:10.9691197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:10.9691520Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:10.9691958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:10.9692315Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:10.9692646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:07:10.9693126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:10.9693503Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:10.9693943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:10.9694279Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:10.9694624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:07:10.9695128Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:07:10.9695655Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:07:10.9696033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:07:10.9696384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:07:10.9696652Z dist init r=0, world=2 2022-11-23T01:07:10.9696823Z dist init r=1, world=2 2022-11-23T01:07:10.9697034Z skip: Need at least 2 CUDA devices (1.613s) 2022-11-23T01:07:10.9697431Z test_simple_model_with_torchdistX_default_init (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s) 2022-11-23T01:07:10.9697943Z test_simple_model_with_torchdistX_init_fn (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s) 2022-11-23T01:07:10.9698195Z 2022-11-23T01:07:10.9698398Z ---------------------------------------------------------------------- 2022-11-23T01:07:10.9698649Z Ran 14 tests in 11.654s 2022-11-23T01:07:10.9698766Z 2022-11-23T01:07:10.9698841Z OK (skipped=14) 2022-11-23T01:07:10.9698952Z 2022-11-23T01:07:10.9699025Z Generating XML reports... 2022-11-23T01:07:10.9699494Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_meta/TEST-TestFSDPWithMetaDevice-20221123010658.xml 2022-11-23T01:07:10.9699769Z 2022-11-23T01:07:10.9700188Z ##[endgroup] 2022-11-23T01:07:10.9700644Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_meta (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_meta_gbzoc299) 2022-11-23T01:07:10.9700903Z 2022-11-23T01:07:10.9701091Z Running distributed/_tensor/test_matrix_ops ... [2022-11-23 01:07:10.962342] 2022-11-23T01:07:10.9701612Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_matrix_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:07:10.962758] 2022-11-23T01:07:26.0931878Z 2022-11-23T01:07:26.0932378Z Expand the folded group to see the log file of distributed/_tensor/test_matrix_ops 2022-11-23T01:07:26.0933397Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_matrix_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_matrix_ops_kcdgpa8r) 2022-11-23T01:07:26.0935125Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_matrix_ops 2022-11-23T01:07:26.0935560Z 2022-11-23T01:07:26.0935702Z Running tests... 2022-11-23T01:07:26.0936109Z ---------------------------------------------------------------------- 2022-11-23T01:07:26.0936642Z test_addmm (__main__.DistMatrixOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3808 2022-11-23T01:07:26.0937020Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3809 2022-11-23T01:07:26.0961896Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3810 2022-11-23T01:07:26.0962503Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3811 2022-11-23T01:07:26.0963292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:26.0963860Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:26.0968274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:26.0968974Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:26.0969522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:07:26.0970301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:26.0970637Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:26.0971074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:26.0971423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:26.0971750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:07:26.0972203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:26.0972539Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:26.0972969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:26.0973310Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:26.0973636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:07:26.0974152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:26.0974731Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:26.0975458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:26.0976008Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:26.0976551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:07:26.0977231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:07:26.0977882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:07:26.0978421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:07:26.0979036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:07:26.0979751Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:26.0980277Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:26.0980798Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:26.0981479Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:26.0981872Z ok (2.173s) 2022-11-23T01:07:26.0982776Z test_addmm_auto_redistribute (__main__.DistMatrixOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3864 2022-11-23T01:07:26.0983631Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3865 2022-11-23T01:07:26.0984345Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3866 2022-11-23T01:07:26.0984941Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3867 2022-11-23T01:07:26.0985682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:26.0986066Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:26.0986599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:26.0986934Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:26.0987262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:07:26.0987723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:26.0988177Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:26.0988763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:26.0989519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:26.0990221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:07:26.0991211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:26.0991925Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:26.0992898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:26.0993431Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:26.0993924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:07:26.0994665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:26.0995274Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:26.0995849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:26.0996197Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:26.0996523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:07:26.0996885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:07:26.0997245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:07:26.0997610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:07:26.0998139Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:26.0998531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:07:26.0999005Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:26.0999521Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:26.1000234Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:26.1000861Z ok (1.820s) 2022-11-23T01:07:26.1001258Z test_baddbmm (__main__.DistMatrixOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3920 2022-11-23T01:07:26.1001636Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3921 2022-11-23T01:07:26.1001970Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3922 2022-11-23T01:07:26.1002281Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3923 2022-11-23T01:07:26.1002743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:26.1003076Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:26.1003564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:26.1003900Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:26.1004230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:07:26.1004692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:26.1005011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:26.1005444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:26.1005791Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:26.1006114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:07:26.1006666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:26.1007001Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:26.1007431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:26.1007766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:26.1008089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:07:26.1008547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:26.1008876Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:26.1009297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:26.1009642Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:26.1009969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:07:26.1010329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:07:26.1010691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:07:26.1011055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:07:26.1011551Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:26.1011934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:07:26.1012423Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:26.1012937Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:26.1013452Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:26.1013760Z skip: Need at least 4 CUDA devices (1.922s) 2022-11-23T01:07:26.1014129Z test_bmm (__main__.DistMatrixOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3976 2022-11-23T01:07:26.1014502Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3977 2022-11-23T01:07:26.1014828Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3978 2022-11-23T01:07:26.1015220Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3979 2022-11-23T01:07:26.1015669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:26.1015988Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:26.1016450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:26.1016796Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:26.1017107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:07:26.1017567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:26.1017895Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:26.1018325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:26.1018657Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:26.1018976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:07:26.1019431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:26.1019751Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:26.1020176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:26.1020524Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:26.1020846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:07:26.1021294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:26.1021623Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:26.1022052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:26.1022398Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:26.1022707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:07:26.1023071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:07:26.1023445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:07:26.1023794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:07:26.1024152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:07:26.1024643Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:26.1025165Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:26.1025667Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:26.1026183Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:26.1026469Z ok (2.221s) 2022-11-23T01:07:26.1026798Z test_mm (__main__.DistMatrixOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4044 2022-11-23T01:07:26.1027154Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4045 2022-11-23T01:07:26.1027481Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4046 2022-11-23T01:07:26.1027806Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4047 2022-11-23T01:07:26.1028243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:26.1028570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:26.1029035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:26.1029380Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:26.1029689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:07:26.1030150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:26.1030477Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:26.1030892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:26.1031236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:26.1031558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:07:26.1032085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:26.1032406Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:26.1032839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:26.1033186Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:26.1033507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:07:26.1033952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:26.1034277Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:26.1034704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:26.1035036Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:26.1035360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:07:26.1035720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:07:26.1036087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:07:26.1036435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:07:26.1036927Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:26.1037323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:07:26.1037812Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:26.1038315Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:26.1038829Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:26.1039116Z ok (1.820s) 2022-11-23T01:07:26.1039436Z test_t (__main__.DistMatrixOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4100 2022-11-23T01:07:26.1039805Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4101 2022-11-23T01:07:26.1040130Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4102 2022-11-23T01:07:26.1040458Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4103 2022-11-23T01:07:26.1041123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:26.1041459Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:26.1041893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:26.1042294Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:26.1042619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:07:26.1043085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:26.1043418Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:26.1043832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:26.1044180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:26.1044504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:07:26.1044965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:26.1045282Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:26.1045712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:26.1046062Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:26.1046371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:07:26.1046825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:26.1047152Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:26.1047577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:26.1047905Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:26.1048227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:07:26.1048590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:07:26.1048943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:07:26.1049308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:07:26.1049803Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:26.1050201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:07:26.1050677Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:26.1051192Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:26.1051710Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:26.1051998Z ok (1.720s) 2022-11-23T01:07:26.1052335Z test_t_partial (__main__.DistMatrixOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4156 2022-11-23T01:07:26.1052711Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4157 2022-11-23T01:07:26.1053040Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4158 2022-11-23T01:07:26.1053348Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4159 2022-11-23T01:07:26.1053802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:26.1054133Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:26.1054563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:26.1054929Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:26.1055252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:07:26.1055717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:26.1056046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:26.1056461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:26.1056807Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:26.1057130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:07:26.1057576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:26.1057907Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:26.1058335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:26.1058683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:26.1058988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:07:26.1059445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:26.1059775Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:26.1060189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:26.1060532Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:26.1060853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:07:26.1061217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:07:26.1061572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:07:26.1061934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:07:26.1062296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:07:26.1062790Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:26.1063291Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:26.1063811Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:26.1064331Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:26.1064616Z ok (1.820s) 2022-11-23T01:07:26.1064706Z 2022-11-23T01:07:26.1064906Z ---------------------------------------------------------------------- 2022-11-23T01:07:26.1065185Z Ran 7 tests in 13.498s 2022-11-23T01:07:26.1065302Z 2022-11-23T01:07:26.1065376Z OK (skipped=1) 2022-11-23T01:07:26.1065483Z 2022-11-23T01:07:26.1065554Z Generating XML reports... 2022-11-23T01:07:26.1066007Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_matrix_ops/TEST-DistMatrixOpsTest-20221123010712.xml 2022-11-23T01:07:26.1066266Z 2022-11-23T01:07:26.1066584Z ##[endgroup] 2022-11-23T01:07:26.1067030Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_matrix_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_matrix_ops_kcdgpa8r) 2022-11-23T01:07:26.1067286Z 2022-11-23T01:07:26.1067556Z Running distributed/fsdp/test_fsdp_sharded_grad_scaler ... [2022-11-23 01:07:26.093824] 2022-11-23T01:07:26.1068112Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_sharded_grad_scaler.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:07:26.094165] 2022-11-23T01:07:42.5490210Z 2022-11-23T01:07:42.5491428Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_sharded_grad_scaler 2022-11-23T01:07:42.5492751Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_sharded_grad_scaler (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_sharded_grad_scaler_1ue0tqr6) 2022-11-23T01:07:42.5493902Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler 2022-11-23T01:07:42.5494284Z 2022-11-23T01:07:42.5494639Z Running tests... 2022-11-23T01:07:42.5495247Z ---------------------------------------------------------------------- 2022-11-23T01:07:42.5495886Z test_grad_scaling (__main__.TestShardGradScaler) ... skip: no supported device (cuda, xla) found (0.001s) 2022-11-23T01:07:42.5496568Z test_inf_gradients_skip_optim_step (__main__.TestShardGradScaler) ... skip: no supported device (cuda, xla) found (0.001s) 2022-11-23T01:07:42.5497204Z test_scaling_unscaling_sparse (__main__.TestShardGradScaler) ... skip: no supported device (cuda, xla) found (0.002s) 2022-11-23T01:07:42.5498078Z test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4223 2022-11-23T01:07:42.5498848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4224 2022-11-23T01:07:42.5499407Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4225 2022-11-23T01:07:42.5499934Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4226 2022-11-23T01:07:42.5500821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:42.5501453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:42.5502212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:42.5502870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:42.5503271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:07:42.5503756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:42.5504091Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:42.5504507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:42.5504858Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:42.5505199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:07:42.5505653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:42.5506180Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:42.5506817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:42.5507258Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:42.5507590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:07:42.5508062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:42.5508392Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:42.5508882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:42.5509228Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:42.5509694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:07:42.5510362Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:42.5510886Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:42.5511461Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:42.5512114Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:42.5512508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:07:42.5512912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:07:42.5513257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:07:42.5513672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:07:42.5513937Z dist init r=1, world=4 2022-11-23T01:07:42.5514110Z dist init r=2, world=4 2022-11-23T01:07:42.5514304Z dist init r=3, world=4 2022-11-23T01:07:42.5514532Z dist init r=0, world=4 2022-11-23T01:07:42.5514726Z skip: Need at least 2 CUDA devices (2.083s) 2022-11-23T01:07:42.5515210Z test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4279 2022-11-23T01:07:42.5515670Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4280 2022-11-23T01:07:42.5516046Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4281 2022-11-23T01:07:42.5516372Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4282 2022-11-23T01:07:42.5516896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:42.5517230Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:42.5517719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:42.5518066Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:42.5518459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:07:42.5518933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:42.5519310Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:42.5519753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:42.5520138Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:42.5520518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:07:42.5521250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:42.5521637Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:42.5522070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:42.5522460Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:42.5522796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:07:42.5523417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:42.5523735Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:42.5524219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:42.5524596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:42.5524955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:07:42.5525504Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:42.5526031Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:42.5526612Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:42.5527187Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:42.5527557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:07:42.5527965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:07:42.5528312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:07:42.5528641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:07:42.5528904Z dist init r=2, world=4 2022-11-23T01:07:42.5529087Z dist init r=3, world=4 2022-11-23T01:07:42.5529319Z dist init r=1, world=4 2022-11-23T01:07:42.5529489Z dist init r=0, world=4 2022-11-23T01:07:42.5529698Z skip: Need at least 2 CUDA devices (1.819s) 2022-11-23T01:07:42.5530147Z test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4335 2022-11-23T01:07:42.5530616Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4336 2022-11-23T01:07:42.5530945Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4337 2022-11-23T01:07:42.5531273Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4338 2022-11-23T01:07:42.5531782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:42.5532100Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:42.5532528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:42.5532881Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:42.5533208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:07:42.5533678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:42.5534064Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:42.5534494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:42.5534826Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:42.5535178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:07:42.5535696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:42.5536026Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:42.5536439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:42.5536820Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:42.5537155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:07:42.5537615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:42.5538019Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:42.5538447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:42.5538794Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:42.5539114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:07:42.5539605Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:42.5540133Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:42.5540657Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:42.5541160Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:42.5541549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:07:42.5541894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:07:42.5542290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:07:42.5542640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:07:42.5542901Z dist init r=3, world=4 2022-11-23T01:07:42.5543085Z dist init r=0, world=4 2022-11-23T01:07:42.5543249Z dist init r=1, world=4 2022-11-23T01:07:42.5543428Z dist init r=2, world=4 2022-11-23T01:07:42.5543640Z skip: Need at least 2 CUDA devices (1.820s) 2022-11-23T01:07:42.5544066Z test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4391 2022-11-23T01:07:42.5544533Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4392 2022-11-23T01:07:42.5544866Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4393 2022-11-23T01:07:42.5545177Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4394 2022-11-23T01:07:42.5545631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:42.5545965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:42.5546397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:42.5546735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:42.5547115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:07:42.5547582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:42.5547909Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:42.5548322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:42.5548668Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:42.5549002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:07:42.5549491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:42.5549816Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:42.5550243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:42.5550587Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:42.5550907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:07:42.5551404Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:42.5551958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:42.5552295Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:42.5552717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:42.5553062Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:42.5553401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:07:42.5553882Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:42.5554403Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:42.5554922Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:42.5555306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:07:42.5555713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:07:42.5556057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:07:42.5556400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:07:42.5556659Z dist init r=2, world=4 2022-11-23T01:07:42.5556829Z dist init r=1, world=4 2022-11-23T01:07:42.5557008Z dist init r=0, world=4 2022-11-23T01:07:42.5557187Z dist init r=3, world=4 2022-11-23T01:07:42.5557377Z skip: Need at least 2 CUDA devices (1.822s) 2022-11-23T01:07:42.5557818Z test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4447 2022-11-23T01:07:42.5558286Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4448 2022-11-23T01:07:42.5558599Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4449 2022-11-23T01:07:42.5558918Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4450 2022-11-23T01:07:42.5559367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:42.5559698Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:42.5560153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:42.5560504Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:42.5560959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:07:42.5561416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:42.5561747Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:42.5562179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:42.5562616Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:42.5562939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:07:42.5563411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:42.5563740Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:42.5564168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:42.5564499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:42.5564835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:07:42.5565331Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:42.5565835Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:42.5566334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:42.5566661Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:42.5567091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:42.5567424Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:42.5567761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:07:42.5568258Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:42.5568779Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:42.5569153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:07:42.5569502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:07:42.5569850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:07:42.5570179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:07:42.5570439Z dist init r=3, world=4 2022-11-23T01:07:42.5570621Z dist init r=0, world=4 2022-11-23T01:07:42.5570802Z dist init r=1, world=4 2022-11-23T01:07:42.5570967Z dist init r=2, world=4 2022-11-23T01:07:42.5571174Z skip: Need at least 2 CUDA devices (1.820s) 2022-11-23T01:07:42.5571603Z test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4503 2022-11-23T01:07:42.5572049Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4504 2022-11-23T01:07:42.5572375Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4505 2022-11-23T01:07:42.5572750Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4506 2022-11-23T01:07:42.5573208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:42.5573527Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:42.5573953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:42.5574297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:42.5574625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:07:42.5575124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:42.5575455Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:42.5575886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:42.5576214Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:42.5576548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:07:42.5577014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:42.5577333Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:42.5577764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:42.5578107Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:42.5578441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:07:42.5578892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:42.5579220Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:42.5579645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:42.5579988Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:42.5580305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:07:42.5580798Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:42.5581320Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:42.5581901Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:42.5582426Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:42.5582814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:07:42.5583161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:07:42.5583495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:07:42.5583833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:07:42.5584091Z dist init r=2, world=4 2022-11-23T01:07:42.5584259Z dist init r=1, world=4 2022-11-23T01:07:42.5584441Z dist init r=0, world=4 2022-11-23T01:07:42.5584622Z dist init r=3, world=4 2022-11-23T01:07:42.5584831Z skip: Need at least 2 CUDA devices (1.720s) 2022-11-23T01:07:42.5585319Z test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4559 2022-11-23T01:07:42.5585794Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4560 2022-11-23T01:07:42.5586122Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4561 2022-11-23T01:07:42.5586435Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4562 2022-11-23T01:07:42.5586889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:42.5587225Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:42.5587658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:42.5588024Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:42.5588362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:07:42.5588831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:42.5589161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:42.5589576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:42.5589917Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:42.5590252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:07:42.5590709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:42.5591039Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:42.5591471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:42.5591820Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:42.5592223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:07:42.5592721Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:42.5593220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:42.5593534Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:42.5593961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:42.5594309Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:42.5594642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:07:42.5595127Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:42.5595649Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:42.5596207Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:42.5596590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:07:42.5596923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:07:42.5597260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:07:42.5597605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:07:42.5597863Z dist init r=0, world=4 2022-11-23T01:07:42.5598030Z dist init r=1, world=4 2022-11-23T01:07:42.5598205Z dist init r=2, world=4 2022-11-23T01:07:42.5598420Z dist init r=3, world=4 2022-11-23T01:07:42.5598611Z skip: Need at least 2 CUDA devices (1.920s) 2022-11-23T01:07:42.5599042Z test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4615 2022-11-23T01:07:42.5599509Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4616 2022-11-23T01:07:42.5599819Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4617 2022-11-23T01:07:42.5600137Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4618 2022-11-23T01:07:42.5600749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:42.5601080Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:42.5601503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:42.5601850Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:42.5602183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:07:42.5602638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:42.5602966Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:42.5603389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:42.5603730Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:42.5604051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:07:42.5604545Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:42.5605039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:42.5605362Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:42.5605773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:42.5606111Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:42.5606445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:07:42.5606898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:42.5607224Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:42.5607647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:42.5607990Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:42.5608310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:07:42.5608805Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:42.5609320Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:42.5609834Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:42.5610201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:07:42.5610542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:07:42.5610885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:07:42.5611271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:07:42.5611532Z dist init r=1, world=4 2022-11-23T01:07:42.5611716Z dist init r=2, world=4 2022-11-23T01:07:42.5611893Z dist init r=3, world=4 2022-11-23T01:07:42.5612055Z dist init r=0, world=4 2022-11-23T01:07:42.5612260Z skip: Need at least 2 CUDA devices (1.820s) 2022-11-23T01:07:42.5612399Z 2022-11-23T01:07:42.5612605Z ---------------------------------------------------------------------- 2022-11-23T01:07:42.5612837Z Ran 11 tests in 14.828s 2022-11-23T01:07:42.5612951Z 2022-11-23T01:07:42.5613023Z OK (skipped=11) 2022-11-23T01:07:42.5613131Z 2022-11-23T01:07:42.5613253Z Generating XML reports... 2022-11-23T01:07:42.5613716Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardGradScaler-20221123010727.xml 2022-11-23T01:07:42.5614400Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardedGradScalerParityWithDDP-20221123010727.xml 2022-11-23T01:07:42.5614719Z 2022-11-23T01:07:42.5615182Z ##[endgroup] 2022-11-23T01:07:42.5615682Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_sharded_grad_scaler (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_sharded_grad_scaler_1ue0tqr6) 2022-11-23T01:07:42.5615964Z 2022-11-23T01:07:42.5616166Z Running distributed/fsdp/test_fsdp_freezing_weights ... [2022-11-23 01:07:42.550534] 2022-11-23T01:07:42.5616728Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_freezing_weights.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:07:42.550909] 2022-11-23T01:07:58.6031380Z 2022-11-23T01:07:58.6032257Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_freezing_weights 2022-11-23T01:07:58.6033425Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_freezing_weights (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_freezing_weights_9q257mjh) 2022-11-23T01:07:58.6034514Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights 2022-11-23T01:07:58.6034921Z 2022-11-23T01:07:58.6035042Z Running tests... 2022-11-23T01:07:58.6035574Z ---------------------------------------------------------------------- 2022-11-23T01:07:58.6036447Z test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4682 2022-11-23T01:07:58.6037319Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4683 2022-11-23T01:07:58.6037898Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4684 2022-11-23T01:07:58.6038458Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4685 2022-11-23T01:07:58.6039177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:58.6039715Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:58.6040457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:58.6041182Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:58.6041740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:07:58.6042520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:58.6043019Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:58.6043702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:58.6044250Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:58.6045012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:07:58.6045703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:58.6046215Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:58.6046810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:58.6047285Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:58.6047763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:07:58.6048599Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:58.6049358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:58.6049856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:58.6050503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:58.6051023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:58.6051498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:07:58.6052182Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:58.6052898Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:58.6053617Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:58.6054146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:07:58.6054599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:07:58.6055063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:07:58.6055543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:07:58.6055895Z dist init r=0, world=4 2022-11-23T01:07:58.6056139Z dist init r=3, world=4 2022-11-23T01:07:58.6056391Z dist init r=1, world=4 2022-11-23T01:07:58.6056630Z dist init r=2, world=4 2022-11-23T01:07:58.6056921Z skip: Need at least 2 CUDA devices (2.084s) 2022-11-23T01:07:58.6057577Z test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4738 2022-11-23T01:07:58.6058272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4739 2022-11-23T01:07:58.6058732Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4740 2022-11-23T01:07:58.6059200Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4741 2022-11-23T01:07:58.6059889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:58.6060346Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:58.6060988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:58.6061481Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:58.6061965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:07:58.6062635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:58.6063196Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:58.6063824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:58.6064325Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:58.6064787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:07:58.6065476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:58.6065947Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:58.6066645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:58.6067139Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:58.6067629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:07:58.6068277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:58.6068737Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:58.6069379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:58.6069855Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:58.6070292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:07:58.6070960Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:58.6071728Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:58.6072560Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:58.6073281Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:58.6073808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:07:58.6074353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:07:58.6074920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:07:58.6075469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:07:58.6075889Z dist init r=3, world=4 2022-11-23T01:07:58.6076197Z dist init r=2, world=4 2022-11-23T01:07:58.6076459Z dist init r=0, world=4 2022-11-23T01:07:58.6076721Z dist init r=1, world=4 2022-11-23T01:07:58.6077042Z skip: Need at least 2 CUDA devices (1.719s) 2022-11-23T01:07:58.6077696Z test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4794 2022-11-23T01:07:58.6078398Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4795 2022-11-23T01:07:58.6078848Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4796 2022-11-23T01:07:58.6079309Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4797 2022-11-23T01:07:58.6079980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:58.6080488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:58.6081344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:58.6082029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:58.6082535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:07:58.6083356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:58.6083918Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:58.6084571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:58.6085071Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:58.6085553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:07:58.6086357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:58.6086799Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:58.6087422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:58.6087914Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:58.6088434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:07:58.6089292Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:58.6090097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:58.6090596Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:58.6091260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:58.6091794Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:58.6092300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:07:58.6093065Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:58.6093900Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:58.6094779Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:58.6095369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:07:58.6095894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:07:58.6096407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:07:58.6096919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:07:58.6097304Z dist init r=1, world=4 2022-11-23T01:07:58.6097568Z dist init r=3, world=4 2022-11-23T01:07:58.6097839Z dist init r=0, world=4 2022-11-23T01:07:58.6098094Z dist init r=2, world=4 2022-11-23T01:07:58.6098383Z skip: Need at least 2 CUDA devices (1.820s) 2022-11-23T01:07:58.6099239Z test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4850 2022-11-23T01:07:58.6100018Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4851 2022-11-23T01:07:58.6100564Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4852 2022-11-23T01:07:58.6101056Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4853 2022-11-23T01:07:58.6101724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:58.6102255Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:58.6102886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:58.6103383Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:58.6103881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:07:58.6104595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:58.6105143Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:58.6105883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:58.6106428Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:58.6106964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:07:58.6107827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:58.6108380Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:58.6109139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:58.6109759Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:58.6110315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:07:58.6111132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:58.6111694Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:58.6112570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:58.6113125Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:58.6113650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:07:58.6114477Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:58.6115358Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:58.6116230Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:58.6117173Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:58.6117859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:07:58.6118430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:07:58.6119036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:07:58.6119603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:07:58.6120033Z dist init r=0, world=4 2022-11-23T01:07:58.6120334Z dist init r=2, world=4 2022-11-23T01:07:58.6120772Z dist init r=3, world=4 2022-11-23T01:07:58.6121075Z dist init r=1, world=4 2022-11-23T01:07:58.6121423Z skip: Need at least 2 CUDA devices (1.720s) 2022-11-23T01:07:58.6122212Z test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4906 2022-11-23T01:07:58.6123094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4907 2022-11-23T01:07:58.6123791Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4908 2022-11-23T01:07:58.6124382Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4909 2022-11-23T01:07:58.6125206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:58.6125791Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:58.6126571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:58.6127202Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:58.6127840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:07:58.6128668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:58.6129230Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:58.6129945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:58.6130549Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:58.6131142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:07:58.6131995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:58.6132559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:58.6133317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:58.6133931Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:58.6134490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:07:58.6135319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:58.6135889Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:58.6136640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:58.6137225Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:58.6137793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:07:58.6138700Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:58.6139630Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:58.6140536Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:58.6141471Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:58.6142165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:07:58.6142781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:07:58.6143397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:07:58.6144003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:07:58.6144449Z dist init r=3, world=4 2022-11-23T01:07:58.6144739Z dist init r=0, world=4 2022-11-23T01:07:58.6145043Z dist init r=2, world=4 2022-11-23T01:07:58.6145352Z dist init r=1, world=4 2022-11-23T01:07:58.6145701Z skip: Need at least 2 CUDA devices (1.719s) 2022-11-23T01:07:58.6146617Z test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4962 2022-11-23T01:07:58.6147453Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4963 2022-11-23T01:07:58.6148028Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4964 2022-11-23T01:07:58.6148593Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4965 2022-11-23T01:07:58.6149421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:58.6150071Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:58.6150842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:58.6151443Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:58.6152110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:07:58.6152973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:58.6153521Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:58.6154263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:58.6154867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:58.6155420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:07:58.6156213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:58.6156734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:58.6157496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:58.6158090Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:58.6158664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:07:58.6159526Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:58.6160433Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:58.6161506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:58.6162058Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:58.6162833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:58.6163464Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:58.6164009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:07:58.6164872Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:58.6165836Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:58.6166514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:07:58.6167101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:07:58.6167696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:07:58.6168302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:07:58.6168867Z dist init r=0, world=4 2022-11-23T01:07:58.6169176Z dist init r=3, world=4 2022-11-23T01:07:58.6169477Z dist init r=1, world=4 2022-11-23T01:07:58.6169760Z dist init r=2, world=4 2022-11-23T01:07:58.6170122Z skip: Need at least 2 CUDA devices (1.820s) 2022-11-23T01:07:58.6170943Z test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5018 2022-11-23T01:07:58.6171768Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5019 2022-11-23T01:07:58.6172334Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5020 2022-11-23T01:07:58.6173021Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5021 2022-11-23T01:07:58.6173865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:58.6174415Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:58.6175279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:58.6175879Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:58.6176440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:07:58.6177279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:58.6177866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:58.6178651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:58.6179240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:58.6179817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:07:58.6180654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:58.6181233Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:58.6181975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:58.6182587Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:58.6183152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:07:58.6184038Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:58.6184968Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:58.6185840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:58.6186405Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:58.6187149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:58.6187748Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:58.6188326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:07:58.6189224Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:58.6190130Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:58.6190798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:07:58.6191517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:07:58.6192204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:07:58.6192799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:07:58.6193229Z dist init r=0, world=4 2022-11-23T01:07:58.6193539Z dist init r=3, world=4 2022-11-23T01:07:58.6193833Z dist init r=1, world=4 2022-11-23T01:07:58.6194137Z dist init r=2, world=4 2022-11-23T01:07:58.6194481Z skip: Need at least 2 CUDA devices (1.719s) 2022-11-23T01:07:58.6195265Z test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5074 2022-11-23T01:07:58.6196220Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5075 2022-11-23T01:07:58.6196818Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5076 2022-11-23T01:07:58.6197404Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5077 2022-11-23T01:07:58.6198240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:58.6198805Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:58.6199594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:58.6200178Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:58.6200840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:07:58.6201650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:58.6202205Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:58.6202945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:58.6203555Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:58.6204150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:07:58.6205004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:58.6205581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:58.6206307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:58.6206909Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:58.6207492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:07:58.6208358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:07:58.6208955Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:07:58.6209728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:07:58.6210341Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:07:58.6210936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:07:58.6211847Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:58.6212804Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:58.6213726Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:58.6214752Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:07:58.6215436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:07:58.6216045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:07:58.6216637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:07:58.6217239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:07:58.6217690Z dist init r=3, world=4 2022-11-23T01:07:58.6218089Z dist init r=1, world=4 2022-11-23T01:07:58.6218379Z dist init r=0, world=4 2022-11-23T01:07:58.6218670Z dist init r=2, world=4 2022-11-23T01:07:58.6219009Z skip: Need at least 2 CUDA devices (1.820s) 2022-11-23T01:07:58.6219255Z 2022-11-23T01:07:58.6219628Z ---------------------------------------------------------------------- 2022-11-23T01:07:58.6220041Z Ran 8 tests in 14.422s 2022-11-23T01:07:58.6220229Z 2022-11-23T01:07:58.6220354Z OK (skipped=8) 2022-11-23T01:07:58.6220519Z 2022-11-23T01:07:58.6220658Z Generating XML reports... 2022-11-23T01:07:58.6221502Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights/TEST-TestFreezingWeights-20221123010743.xml 2022-11-23T01:07:58.6221986Z 2022-11-23T01:07:58.6222468Z ##[endgroup] 2022-11-23T01:07:58.6223342Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_freezing_weights (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_freezing_weights_9q257mjh) 2022-11-23T01:07:58.6224023Z 2022-11-23T01:07:58.6224374Z Running distributed/test_c10d_spawn_gloo ... [2022-11-23 01:07:58.603859] 2022-11-23T01:07:58.6225278Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_spawn_gloo.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:07:58.604304] 2022-11-23T01:08:43.1666955Z 2022-11-23T01:08:43.1667495Z Expand the folded group to see the log file of distributed/test_c10d_spawn_gloo 2022-11-23T01:08:43.1668628Z ##[group]PRINTING LOG FILE of distributed/test_c10d_spawn_gloo (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_gloo_zk5o_y8s) 2022-11-23T01:08:43.1672229Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp008ipi3v 2022-11-23T01:08:43.1672960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp008ipi3v/_remote_module_non_scriptable.py 2022-11-23T01:08:43.1673615Z , <__main__.DistributedDataParallelSingleProcessTest testMethod=test_cuda>, <__main__.DistributedDataParallelSingleProcessTest testMethod=test_rnn>]> 2022-11-23T01:08:43.1674256Z test_cpu (__main__.DistributedDataParallelSingleProcessTest) 2022-11-23T01:08:43.1674654Z test_cuda (__main__.DistributedDataParallelSingleProcessTest) 2022-11-23T01:08:43.1682722Z test_rnn (__main__.DistributedDataParallelSingleProcessTest) 2022-11-23T01:08:43.1684746Z , <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allgather_gloo>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allreduce_gloo>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_broadcast_gloo>]> 2022-11-23T01:08:43.1685955Z test_shared_allgather_chunk_gloo (__main__.ProcessGroupShareTensorTest) 2022-11-23T01:08:43.1686389Z test_shared_allgather_gloo (__main__.ProcessGroupShareTensorTest) 2022-11-23T01:08:43.1686826Z test_shared_allreduce_gloo (__main__.ProcessGroupShareTensorTest) 2022-11-23T01:08:43.1687257Z test_shared_broadcast_gloo (__main__.ProcessGroupShareTensorTest) 2022-11-23T01:08:43.1687607Z 2022-11-23T01:08:43.1687930Z 2022-11-23T01:08:43.1689353Z , <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_gather>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_scatter>]> 2022-11-23T01:08:43.1690583Z test_all_gather (__main__.TestDistributedNNFunctionsGloo) 2022-11-23T01:08:43.1691088Z test_all_to_all (__main__.TestDistributedNNFunctionsGloo) 2022-11-23T01:08:43.1691499Z test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo) 2022-11-23T01:08:43.1691939Z test_allreduce (__main__.TestDistributedNNFunctionsGloo) 2022-11-23T01:08:43.1692362Z test_broadcast (__main__.TestDistributedNNFunctionsGloo) 2022-11-23T01:08:43.1692786Z test_gather (__main__.TestDistributedNNFunctionsGloo) 2022-11-23T01:08:43.1693195Z test_reduce (__main__.TestDistributedNNFunctionsGloo) 2022-11-23T01:08:43.1693613Z test_scatter (__main__.TestDistributedNNFunctionsGloo) 2022-11-23T01:08:43.1694361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:08:43.1694817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:08:43.1695467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:08:43.1695968Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:08:43.1696467Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptxmyuzn7 2022-11-23T01:08:43.1697024Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptxmyuzn7/_remote_module_non_scriptable.py 2022-11-23T01:08:43.1697700Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T01:08:43.1698036Z 2022-11-23T01:08:43.1698143Z Running tests... 2022-11-23T01:08:43.1698586Z ---------------------------------------------------------------------- 2022-11-23T01:08:43.1699197Z test_cpu (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T01:08:43.1699707Z ok (0.012s) 2022-11-23T01:08:43.1699854Z 2022-11-23T01:08:43.1700157Z ---------------------------------------------------------------------- 2022-11-23T01:08:43.1700523Z Ran 1 test in 0.012s 2022-11-23T01:08:43.1700698Z 2022-11-23T01:08:43.1700765Z OK 2022-11-23T01:08:43.1700905Z 2022-11-23T01:08:43.1701033Z Generating XML reports... 2022-11-23T01:08:43.1701810Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20221123010801.xml 2022-11-23T01:08:43.1702641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:08:43.1703107Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:08:43.1703734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:08:43.1704213Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:08:43.1704663Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpofrvrn_f 2022-11-23T01:08:43.1705203Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpofrvrn_f/_remote_module_non_scriptable.py 2022-11-23T01:08:43.1705888Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T01:08:43.1706157Z 2022-11-23T01:08:43.1706264Z Running tests... 2022-11-23T01:08:43.1706676Z ---------------------------------------------------------------------- 2022-11-23T01:08:43.1707273Z test_cuda (__main__.DistributedDataParallelSingleProcessTest) ... skip: At least 1 CUDA GPUS needed (0.000s) 2022-11-23T01:08:43.1707597Z 2022-11-23T01:08:43.1707884Z ---------------------------------------------------------------------- 2022-11-23T01:08:43.1708220Z Ran 1 test in 0.001s 2022-11-23T01:08:43.1708397Z 2022-11-23T01:08:43.1708518Z OK (skipped=1) 2022-11-23T01:08:43.1708692Z 2022-11-23T01:08:43.1708838Z Generating XML reports... 2022-11-23T01:08:43.1709716Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20221123010803.xml 2022-11-23T01:08:43.1710715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:08:43.1711388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:08:43.1712161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:08:43.1712793Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:08:43.1713418Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ghb3mxt 2022-11-23T01:08:43.1713934Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ghb3mxt/_remote_module_non_scriptable.py 2022-11-23T01:08:43.1714398Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T01:08:43.1714598Z 2022-11-23T01:08:43.1714658Z Running tests... 2022-11-23T01:08:43.1714961Z ---------------------------------------------------------------------- 2022-11-23T01:08:43.1715349Z test_rnn (__main__.DistributedDataParallelSingleProcessTest) ... skip: At least 1 CUDA GPUS needed (0.001s) 2022-11-23T01:08:43.1715579Z 2022-11-23T01:08:43.1715772Z ---------------------------------------------------------------------- 2022-11-23T01:08:43.1716001Z Ran 1 test in 0.001s 2022-11-23T01:08:43.1716114Z 2022-11-23T01:08:43.1716189Z OK (skipped=1) 2022-11-23T01:08:43.1716300Z 2022-11-23T01:08:43.1716383Z Generating XML reports... 2022-11-23T01:08:43.1716897Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20221123010805.xml 2022-11-23T01:08:43.1717489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:08:43.1717821Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:08:43.1718248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:08:43.1718584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:08:43.1718929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_pu3qm_r 2022-11-23T01:08:43.1719333Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_pu3qm_r/_remote_module_non_scriptable.py 2022-11-23T01:08:43.1719790Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T01:08:43.1719974Z 2022-11-23T01:08:43.1720047Z Running tests... 2022-11-23T01:08:43.1720348Z ---------------------------------------------------------------------- 2022-11-23T01:08:43.1720909Z test_shared_allgather_chunk_gloo (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s) 2022-11-23T01:08:43.1721138Z 2022-11-23T01:08:43.1721322Z ---------------------------------------------------------------------- 2022-11-23T01:08:43.1721562Z Ran 1 test in 0.001s 2022-11-23T01:08:43.1721677Z 2022-11-23T01:08:43.1721753Z OK (skipped=1) 2022-11-23T01:08:43.1721861Z 2022-11-23T01:08:43.1721946Z Generating XML reports... 2022-11-23T01:08:43.1722408Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20221123010807.xml 2022-11-23T01:08:43.1723062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:08:43.1723405Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:08:43.1723824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:08:43.1724174Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:08:43.1724518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnja6nzjq 2022-11-23T01:08:43.1724925Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnja6nzjq/_remote_module_non_scriptable.py 2022-11-23T01:08:43.1725413Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T01:08:43.1725611Z 2022-11-23T01:08:43.1725685Z Running tests... 2022-11-23T01:08:43.1725987Z ---------------------------------------------------------------------- 2022-11-23T01:08:43.1726339Z test_shared_allgather_gloo (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s) 2022-11-23T01:08:43.1726557Z 2022-11-23T01:08:43.1726750Z ---------------------------------------------------------------------- 2022-11-23T01:08:43.1726989Z Ran 1 test in 0.001s 2022-11-23T01:08:43.1727106Z 2022-11-23T01:08:43.1727179Z OK (skipped=1) 2022-11-23T01:08:43.1727286Z 2022-11-23T01:08:43.1727356Z Generating XML reports... 2022-11-23T01:08:43.1727834Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20221123010808.xml 2022-11-23T01:08:43.1728534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:08:43.1728872Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:08:43.1729293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:08:43.1729644Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:08:43.1729991Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp38m04907 2022-11-23T01:08:43.1730378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp38m04907/_remote_module_non_scriptable.py 2022-11-23T01:08:43.1730828Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T01:08:43.1731026Z 2022-11-23T01:08:43.1731099Z Running tests... 2022-11-23T01:08:43.1731401Z ---------------------------------------------------------------------- 2022-11-23T01:08:43.1731750Z test_shared_allreduce_gloo (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s) 2022-11-23T01:08:43.1731970Z 2022-11-23T01:08:43.1732163Z ---------------------------------------------------------------------- 2022-11-23T01:08:43.1732405Z Ran 1 test in 0.001s 2022-11-23T01:08:43.1732519Z 2022-11-23T01:08:43.1732590Z OK (skipped=1) 2022-11-23T01:08:43.1732685Z 2022-11-23T01:08:43.1732772Z Generating XML reports... 2022-11-23T01:08:43.1733251Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20221123010810.xml 2022-11-23T01:08:43.1733803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:08:43.1734125Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:08:43.1734642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:08:43.1735230Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:08:43.1735604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd5wx0wf9 2022-11-23T01:08:43.1735994Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd5wx0wf9/_remote_module_non_scriptable.py 2022-11-23T01:08:43.1736502Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T01:08:43.1736702Z 2022-11-23T01:08:43.1736778Z Running tests... 2022-11-23T01:08:43.1737066Z ---------------------------------------------------------------------- 2022-11-23T01:08:43.1737434Z test_shared_broadcast_gloo (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s) 2022-11-23T01:08:43.1737656Z 2022-11-23T01:08:43.1737850Z ---------------------------------------------------------------------- 2022-11-23T01:08:43.1738093Z Ran 1 test in 0.001s 2022-11-23T01:08:43.1738190Z 2022-11-23T01:08:43.1738264Z OK (skipped=1) 2022-11-23T01:08:43.1738374Z 2022-11-23T01:08:43.1738498Z Generating XML reports... 2022-11-23T01:08:43.1738975Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20221123010812.xml 2022-11-23T01:08:43.1739510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:08:43.1739845Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:08:43.1740281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:08:43.1740629Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:08:43.1740959Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3vnoy288 2022-11-23T01:08:43.1741365Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3vnoy288/_remote_module_non_scriptable.py 2022-11-23T01:08:43.1741819Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T01:08:43.1742021Z 2022-11-23T01:08:43.1742093Z Running tests... 2022-11-23T01:08:43.1742382Z ---------------------------------------------------------------------- 2022-11-23T01:08:43.1742785Z test_all_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5224 2022-11-23T01:08:43.1743194Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5225 2022-11-23T01:08:43.1743632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:08:43.1743963Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:08:43.1744390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:08:43.1744738Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:08:43.1745068Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsta9szrp 2022-11-23T01:08:43.1745482Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsta9szrp/_remote_module_non_scriptable.py 2022-11-23T01:08:43.1745969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:08:43.1746285Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:08:43.1746713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:08:43.1747060Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:08:43.1747404Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbogsp0a9 2022-11-23T01:08:43.1747795Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbogsp0a9/_remote_module_non_scriptable.py 2022-11-23T01:08:43.1748175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:08:43.1748531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:08:43.1748822Z skip: Need at least 2 CUDA devices (1.925s) 2022-11-23T01:08:43.1748946Z 2022-11-23T01:08:43.1749180Z ---------------------------------------------------------------------- 2022-11-23T01:08:43.1749423Z Ran 1 test in 1.925s 2022-11-23T01:08:43.1749540Z 2022-11-23T01:08:43.1749612Z OK (skipped=1) 2022-11-23T01:08:43.1749719Z 2022-11-23T01:08:43.1749792Z Generating XML reports... 2022-11-23T01:08:43.1750285Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123010814.xml 2022-11-23T01:08:43.1750853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:08:43.1751194Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:08:43.1751611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:08:43.1752071Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:08:43.1752423Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpghjul_fp 2022-11-23T01:08:43.1752818Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpghjul_fp/_remote_module_non_scriptable.py 2022-11-23T01:08:43.1753281Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T01:08:43.1753480Z 2022-11-23T01:08:43.1753553Z Running tests... 2022-11-23T01:08:43.1753856Z ---------------------------------------------------------------------- 2022-11-23T01:08:43.1754244Z test_all_to_all (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5257 2022-11-23T01:08:43.1754652Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5258 2022-11-23T01:08:43.1755117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:08:43.1755447Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:08:43.1755866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:08:43.1756217Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:08:43.1756566Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9zfc3r9n 2022-11-23T01:08:43.1756955Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9zfc3r9n/_remote_module_non_scriptable.py 2022-11-23T01:08:43.1757449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:08:43.1757782Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:08:43.1758217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:08:43.1758550Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:08:43.1758895Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcs_psf7_ 2022-11-23T01:08:43.1759304Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcs_psf7_/_remote_module_non_scriptable.py 2022-11-23T01:08:43.1759668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:08:43.1760020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:08:43.1760306Z skip: Need at least 2 CUDA devices (1.924s) 2022-11-23T01:08:43.1760444Z 2022-11-23T01:08:43.1760801Z ---------------------------------------------------------------------- 2022-11-23T01:08:43.1761035Z Ran 1 test in 1.925s 2022-11-23T01:08:43.1761149Z 2022-11-23T01:08:43.1761224Z OK (skipped=1) 2022-11-23T01:08:43.1761337Z 2022-11-23T01:08:43.1761423Z Generating XML reports... 2022-11-23T01:08:43.1761902Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123010817.xml 2022-11-23T01:08:43.1762533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:08:43.1762869Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:08:43.1763303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:08:43.1763637Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:08:43.1763980Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbo2i549j 2022-11-23T01:08:43.1764384Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbo2i549j/_remote_module_non_scriptable.py 2022-11-23T01:08:43.1764899Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T01:08:43.1765084Z 2022-11-23T01:08:43.1765158Z Running tests... 2022-11-23T01:08:43.1765461Z ---------------------------------------------------------------------- 2022-11-23T01:08:43.1765873Z test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5290 2022-11-23T01:08:43.1766268Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5291 2022-11-23T01:08:43.1766723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:08:43.1767055Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:08:43.1767484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:08:43.1767819Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:08:43.1768169Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ddil3c7 2022-11-23T01:08:43.1768578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ddil3c7/_remote_module_non_scriptable.py 2022-11-23T01:08:43.1769051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:08:43.1769383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:08:43.1769811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:08:43.1770156Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:08:43.1770492Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp78r2r4m7 2022-11-23T01:08:43.1770893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp78r2r4m7/_remote_module_non_scriptable.py 2022-11-23T01:08:43.1771275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:08:43.1771622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:08:43.1771896Z skip: Need at least 2 CUDA devices (1.924s) 2022-11-23T01:08:43.1772032Z 2022-11-23T01:08:43.1772233Z ---------------------------------------------------------------------- 2022-11-23T01:08:43.1772473Z Ran 1 test in 1.925s 2022-11-23T01:08:43.1772584Z 2022-11-23T01:08:43.1772643Z OK (skipped=1) 2022-11-23T01:08:43.1772749Z 2022-11-23T01:08:43.1772834Z Generating XML reports... 2022-11-23T01:08:43.1773419Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123010821.xml 2022-11-23T01:08:43.1774266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:08:43.1774685Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:08:43.1775434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:08:43.1775903Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:08:43.1776432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpldmvixid 2022-11-23T01:08:43.1777053Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpldmvixid/_remote_module_non_scriptable.py 2022-11-23T01:08:43.1777763Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T01:08:43.1778051Z 2022-11-23T01:08:43.1778152Z Running tests... 2022-11-23T01:08:43.1778578Z ---------------------------------------------------------------------- 2022-11-23T01:08:43.1779187Z test_allreduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5323 2022-11-23T01:08:43.1779929Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5324 2022-11-23T01:08:43.1780700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:08:43.1781252Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:08:43.1781946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:08:43.1782470Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:08:43.1783059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7eb82e2u 2022-11-23T01:08:43.1783731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7eb82e2u/_remote_module_non_scriptable.py 2022-11-23T01:08:43.1784542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:08:43.1784969Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:08:43.1785612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:08:43.1786098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:08:43.1786618Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptimlvcy_ 2022-11-23T01:08:43.1787173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptimlvcy_/_remote_module_non_scriptable.py 2022-11-23T01:08:43.1787721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:08:43.1788275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:08:43.1788726Z skip: Need at least 2 CUDA devices (1.925s) 2022-11-23T01:08:43.1788944Z 2022-11-23T01:08:43.1789296Z ---------------------------------------------------------------------- 2022-11-23T01:08:43.1789717Z Ran 1 test in 1.925s 2022-11-23T01:08:43.1789999Z 2022-11-23T01:08:43.1790127Z OK (skipped=1) 2022-11-23T01:08:43.1790434Z 2022-11-23T01:08:43.1790591Z Generating XML reports... 2022-11-23T01:08:43.1791464Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123010825.xml 2022-11-23T01:08:43.1792515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:08:43.1793073Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:08:43.1793767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:08:43.1794281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:08:43.1794871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph2tnww2o 2022-11-23T01:08:43.1795566Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph2tnww2o/_remote_module_non_scriptable.py 2022-11-23T01:08:43.1796369Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T01:08:43.1796691Z 2022-11-23T01:08:43.1796805Z Running tests... 2022-11-23T01:08:43.1797301Z ---------------------------------------------------------------------- 2022-11-23T01:08:43.1798088Z test_broadcast (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5356 2022-11-23T01:08:43.1798802Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5357 2022-11-23T01:08:43.1799585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:08:43.1800195Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:08:43.1801135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:08:43.1801895Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:08:43.1802470Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmgi3md2l 2022-11-23T01:08:43.1803177Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmgi3md2l/_remote_module_non_scriptable.py 2022-11-23T01:08:43.1804083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:08:43.1804669Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:08:43.1805442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:08:43.1806091Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:08:43.1806741Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzib942wp 2022-11-23T01:08:43.1807549Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzib942wp/_remote_module_non_scriptable.py 2022-11-23T01:08:43.1808246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:08:43.1808890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:08:43.1809398Z skip: Need at least 2 CUDA devices (1.924s) 2022-11-23T01:08:43.1809637Z 2022-11-23T01:08:43.1810009Z ---------------------------------------------------------------------- 2022-11-23T01:08:43.1810386Z Ran 1 test in 1.925s 2022-11-23T01:08:43.1810581Z 2022-11-23T01:08:43.1810705Z OK (skipped=1) 2022-11-23T01:08:43.1810894Z 2022-11-23T01:08:43.1811031Z Generating XML reports... 2022-11-23T01:08:43.1811892Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123010829.xml 2022-11-23T01:08:43.1812910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:08:43.1813532Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:08:43.1814365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:08:43.1814984Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:08:43.1815617Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprq8ru2ls 2022-11-23T01:08:43.1816331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprq8ru2ls/_remote_module_non_scriptable.py 2022-11-23T01:08:43.1817158Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T01:08:43.1817465Z 2022-11-23T01:08:43.1817581Z Running tests... 2022-11-23T01:08:43.1818067Z ---------------------------------------------------------------------- 2022-11-23T01:08:43.1818726Z test_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5389 2022-11-23T01:08:43.1819379Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5390 2022-11-23T01:08:43.1820140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:08:43.1821174Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:08:43.1821929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:08:43.1822516Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:08:43.1823121Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp_lwi9ou 2022-11-23T01:08:43.1823832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp_lwi9ou/_remote_module_non_scriptable.py 2022-11-23T01:08:43.1824693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:08:43.1825350Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:08:43.1826154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:08:43.1826768Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:08:43.1827376Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmcmxowc4 2022-11-23T01:08:43.1828083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmcmxowc4/_remote_module_non_scriptable.py 2022-11-23T01:08:43.1828756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:08:43.1829374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:08:43.1829868Z skip: Need at least 2 CUDA devices (1.925s) 2022-11-23T01:08:43.1830099Z 2022-11-23T01:08:43.1830498Z ---------------------------------------------------------------------- 2022-11-23T01:08:43.1830931Z Ran 1 test in 1.926s 2022-11-23T01:08:43.1831132Z 2022-11-23T01:08:43.1831262Z OK (skipped=1) 2022-11-23T01:08:43.1831430Z 2022-11-23T01:08:43.1831585Z Generating XML reports... 2022-11-23T01:08:43.1832570Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123010832.xml 2022-11-23T01:08:43.1833588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:08:43.1834184Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:08:43.1834961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:08:43.1835594Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:08:43.1836210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_v6o16et 2022-11-23T01:08:43.1836893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_v6o16et/_remote_module_non_scriptable.py 2022-11-23T01:08:43.1837743Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T01:08:43.1838112Z 2022-11-23T01:08:43.1838242Z Running tests... 2022-11-23T01:08:43.1838798Z ---------------------------------------------------------------------- 2022-11-23T01:08:43.1839482Z test_reduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5422 2022-11-23T01:08:43.1840188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5423 2022-11-23T01:08:43.1841113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:08:43.1841675Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:08:43.1842470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:08:43.1843101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:08:43.1843736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvpk0gi9m 2022-11-23T01:08:43.1844579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvpk0gi9m/_remote_module_non_scriptable.py 2022-11-23T01:08:43.1845503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:08:43.1846103Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:08:43.1846892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:08:43.1847512Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:08:43.1848134Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpku59ddh_ 2022-11-23T01:08:43.1848950Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpku59ddh_/_remote_module_non_scriptable.py 2022-11-23T01:08:43.1849614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:08:43.1850241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:08:43.1850759Z skip: Need at least 2 CUDA devices (1.924s) 2022-11-23T01:08:43.1851000Z 2022-11-23T01:08:43.1851373Z ---------------------------------------------------------------------- 2022-11-23T01:08:43.1851784Z Ran 1 test in 1.925s 2022-11-23T01:08:43.1851981Z 2022-11-23T01:08:43.1852108Z OK (skipped=1) 2022-11-23T01:08:43.1852291Z 2022-11-23T01:08:43.1852441Z Generating XML reports... 2022-11-23T01:08:43.1853307Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123010836.xml 2022-11-23T01:08:43.1854343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:08:43.1854959Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:08:43.1855728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:08:43.1856328Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:08:43.1856951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf8jutxzs 2022-11-23T01:08:43.1857662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf8jutxzs/_remote_module_non_scriptable.py 2022-11-23T01:08:43.1858506Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T01:08:43.1858855Z 2022-11-23T01:08:43.1858981Z Running tests... 2022-11-23T01:08:43.1859534Z ---------------------------------------------------------------------- 2022-11-23T01:08:43.1860256Z test_scatter (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5455 2022-11-23T01:08:43.1860985Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5456 2022-11-23T01:08:43.1861801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:08:43.1862372Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:08:43.1863178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:08:43.1863766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:08:43.1864372Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc_44mals 2022-11-23T01:08:43.1865070Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc_44mals/_remote_module_non_scriptable.py 2022-11-23T01:08:43.1865960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:08:43.1866568Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:08:43.1867365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:08:43.1868091Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:08:43.1868659Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjtopn561 2022-11-23T01:08:43.1869399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjtopn561/_remote_module_non_scriptable.py 2022-11-23T01:08:43.1870091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:08:43.1870728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:08:43.1871231Z skip: Need at least 2 CUDA devices (1.926s) 2022-11-23T01:08:43.1871480Z 2022-11-23T01:08:43.1871946Z ---------------------------------------------------------------------- 2022-11-23T01:08:43.1872474Z Ran 1 test in 1.926s 2022-11-23T01:08:43.1872654Z 2022-11-23T01:08:43.1872783Z OK (skipped=1) 2022-11-23T01:08:43.1872964Z 2022-11-23T01:08:43.1873113Z Generating XML reports... 2022-11-23T01:08:43.1874012Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123010840.xml 2022-11-23T01:08:43.1874481Z 2022-11-23T01:08:43.1874890Z ##[endgroup] 2022-11-23T01:08:43.1875612Z FINISHED PRINTING LOG FILE of distributed/test_c10d_spawn_gloo (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_gloo_zk5o_y8s) 2022-11-23T01:08:43.1876012Z 2022-11-23T01:08:43.1876379Z Running distributed/_tensor/parallel/test_tp_style ... [2022-11-23 01:08:43.167438] 2022-11-23T01:08:43.1877288Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/parallel/test_tp_style.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:08:43.167768] 2022-11-23T01:09:00.7170354Z 2022-11-23T01:09:00.7171007Z Expand the folded group to see the log file of distributed/_tensor/parallel/test_tp_style 2022-11-23T01:09:00.7172436Z ##[group]PRINTING LOG FILE of distributed/_tensor/parallel/test_tp_style (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-parallel-test_tp_style_joedliwc) 2022-11-23T01:09:00.7173490Z Test results will be stored in test-reports/python-unittest/distributed._tensor.parallel.test_tp_style 2022-11-23T01:09:00.7173865Z 2022-11-23T01:09:00.7173967Z Running tests... 2022-11-23T01:09:00.7174500Z ---------------------------------------------------------------------- 2022-11-23T01:09:00.7175220Z test_colwise_parallel_style (__main__.TensorParallelStyleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5488 2022-11-23T01:09:00.7175952Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5489 2022-11-23T01:09:00.7176533Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5490 2022-11-23T01:09:00.7177100Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5491 2022-11-23T01:09:00.7188279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:09:00.7188888Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:09:00.7189592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:09:00.7189940Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:09:00.7190466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:09:00.7190916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:09:00.7191245Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:09:00.7191679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:09:00.7192095Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:09:00.7192407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:09:00.7193064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:09:00.7193396Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:09:00.7193824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:09:00.7194156Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:09:00.7194479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:09:00.7194936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:09:00.7195310Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:09:00.7195740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:09:00.7196090Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:09:00.7196412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:09:00.7196762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:09:00.7197126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:09:00.7197491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:09:00.7197837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:09:00.7198336Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:09:00.7198858Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:09:00.7199377Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:09:00.7199877Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:09:00.7200267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T01:09:00.7200793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2022-11-23T01:09:00.7201157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-11-23T01:09:00.7201506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T01:09:00.7201993Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T01:09:00.7202511Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T01:09:00.7202904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T01:09:00.7203379Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T01:09:00.7203888Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T01:09:00.7204278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2 2022-11-23T01:09:00.7204645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3 2022-11-23T01:09:00.7204991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T01:09:00.7205531Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes. 2022-11-23T01:09:00.7206046Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes. 2022-11-23T01:09:00.7206545Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes. 2022-11-23T01:09:00.7207056Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes. 2022-11-23T01:09:00.7207448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2 2022-11-23T01:09:00.7207807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3 2022-11-23T01:09:00.7208193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T01:09:00.7208549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T01:09:00.7209035Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes. 2022-11-23T01:09:00.7209429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0 2022-11-23T01:09:00.7209907Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes. 2022-11-23T01:09:00.7210415Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes. 2022-11-23T01:09:00.7210932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes. 2022-11-23T01:09:00.7211319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1 2022-11-23T01:09:00.7211665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3 2022-11-23T01:09:00.7212029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2 2022-11-23T01:09:00.7212513Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes. 2022-11-23T01:09:00.7213022Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes. 2022-11-23T01:09:00.7213518Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes. 2022-11-23T01:09:00.7214026Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes. 2022-11-23T01:09:00.7214315Z ok (2.175s) 2022-11-23T01:09:00.7214636Z test_make_input_replicate_1d (__main__.TensorParallelStyleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5568 2022-11-23T01:09:00.7215039Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5569 2022-11-23T01:09:00.7215366Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5570 2022-11-23T01:09:00.7215690Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5571 2022-11-23T01:09:00.7216128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:09:00.7216463Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:09:00.7216893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:09:00.7217239Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:09:00.7217554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:09:00.7218013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:09:00.7218369Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:09:00.7218786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:09:00.7219129Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:09:00.7219451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:09:00.7219907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:09:00.7220223Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:09:00.7220682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:09:00.7221025Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:09:00.7221332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:09:00.7221792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:09:00.7222118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:09:00.7222543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:09:00.7222873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:09:00.7223194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:09:00.7223560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:09:00.7223929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:09:00.7224278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:09:00.7224640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:09:00.7225130Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:09:00.7225634Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:09:00.7226146Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:09:00.7226662Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:09:00.7227059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T01:09:00.7227410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2022-11-23T01:09:00.7227770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-11-23T01:09:00.7228129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T01:09:00.7228608Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T01:09:00.7228989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T01:09:00.7229476Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T01:09:00.7229983Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T01:09:00.7230500Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T01:09:00.7230909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2 2022-11-23T01:09:00.7231271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3 2022-11-23T01:09:00.7231629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T01:09:00.7232179Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes. 2022-11-23T01:09:00.7232689Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes. 2022-11-23T01:09:00.7233201Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes. 2022-11-23T01:09:00.7233760Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes. 2022-11-23T01:09:00.7234136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2 2022-11-23T01:09:00.7234494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3 2022-11-23T01:09:00.7234853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T01:09:00.7235216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T01:09:00.7235683Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes. 2022-11-23T01:09:00.7236195Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes. 2022-11-23T01:09:00.7236590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0 2022-11-23T01:09:00.7237074Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes. 2022-11-23T01:09:00.7237576Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes. 2022-11-23T01:09:00.7237965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1 2022-11-23T01:09:00.7238327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3 2022-11-23T01:09:00.7238688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2 2022-11-23T01:09:00.7239159Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes. 2022-11-23T01:09:00.7239671Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes. 2022-11-23T01:09:00.7240180Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes. 2022-11-23T01:09:00.7240775Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes. 2022-11-23T01:09:00.7241047Z ok (1.919s) 2022-11-23T01:09:00.7241373Z test_make_input_shard_1d (__main__.TensorParallelStyleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5648 2022-11-23T01:09:00.7241770Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5649 2022-11-23T01:09:00.7242082Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5650 2022-11-23T01:09:00.7242399Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5651 2022-11-23T01:09:00.7242856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:09:00.7243190Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:09:00.7243665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:09:00.7244019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:09:00.7244342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:09:00.7244791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:09:00.7245123Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:09:00.7245549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:09:00.7245896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:09:00.7246247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:09:00.7246705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:09:00.7247037Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:09:00.7247451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:09:00.7247800Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:09:00.7248118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:09:00.7248576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:09:00.7248889Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:09:00.7249316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:09:00.7249659Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:09:00.7249979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:09:00.7250328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:09:00.7250690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:09:00.7251180Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:09:00.7251562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:09:00.7251924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:09:00.7252418Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:09:00.7252936Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:09:00.7253440Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:09:00.7253830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T01:09:00.7254193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2022-11-23T01:09:00.7254553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-11-23T01:09:00.7254900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T01:09:00.7255385Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T01:09:00.7255900Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T01:09:00.7256327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T01:09:00.7256802Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T01:09:00.7257309Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T01:09:00.7257699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2 2022-11-23T01:09:00.7258059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3 2022-11-23T01:09:00.7258401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T01:09:00.7258927Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes. 2022-11-23T01:09:00.7259436Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes. 2022-11-23T01:09:00.7259938Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes. 2022-11-23T01:09:00.7260440Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes. 2022-11-23T01:09:00.7260834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2 2022-11-23T01:09:00.7261195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3 2022-11-23T01:09:00.7261545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T01:09:00.7261906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T01:09:00.7262390Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes. 2022-11-23T01:09:00.7262904Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes. 2022-11-23T01:09:00.7263278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0 2022-11-23T01:09:00.7263765Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes. 2022-11-23T01:09:00.7264269Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes. 2022-11-23T01:09:00.7264658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1 2022-11-23T01:09:00.7265008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2 2022-11-23T01:09:00.7265365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3 2022-11-23T01:09:00.7265853Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes. 2022-11-23T01:09:00.7266362Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes. 2022-11-23T01:09:00.7266853Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes. 2022-11-23T01:09:00.7267362Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes. 2022-11-23T01:09:00.7267648Z ok (1.919s) 2022-11-23T01:09:00.7267971Z test_make_output_replicate_1d (__main__.TensorParallelStyleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5728 2022-11-23T01:09:00.7268378Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5729 2022-11-23T01:09:00.7268705Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5730 2022-11-23T01:09:00.7269097Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5731 2022-11-23T01:09:00.7269536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:09:00.7269866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:09:00.7270298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:09:00.7270631Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:09:00.7270951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:09:00.7271462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:09:00.7271793Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:09:00.7272292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:09:00.7272640Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:09:00.7272964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:09:00.7273420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:09:00.7273735Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:09:00.7274159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:09:00.7274507Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:09:00.7274822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:09:00.7275278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:09:00.7275607Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:09:00.7276033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:09:00.7276364Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:09:00.7276686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:09:00.7277049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:09:00.7277404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:09:00.7277770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:09:00.7278261Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:09:00.7278778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:09:00.7279159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:09:00.7279642Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:09:00.7280154Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:09:00.7280441Z ok (1.820s) 2022-11-23T01:09:00.7280872Z test_make_output_shard_1d (__main__.TensorParallelStyleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5784 2022-11-23T01:09:00.7281273Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5785 2022-11-23T01:09:00.7281597Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5786 2022-11-23T01:09:00.7281970Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5787 2022-11-23T01:09:00.7282426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:09:00.7282757Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:09:00.7283189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:09:00.7283521Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:09:00.7283842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:09:00.7284347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:09:00.7284678Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:09:00.7285096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:09:00.7285440Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:09:00.7285765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:09:00.7286209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:09:00.7286535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:09:00.7286965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:09:00.7287307Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:09:00.7287618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:09:00.7288077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:09:00.7288409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:09:00.7288825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:09:00.7289168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:09:00.7289488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:09:00.7289849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:09:00.7290200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:09:00.7290562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:09:00.7290924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:09:00.7291416Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:09:00.7291920Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:09:00.7292436Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:09:00.7292943Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:09:00.7293229Z ok (1.920s) 2022-11-23T01:09:00.7293543Z test_make_output_tensor (__main__.TensorParallelStyleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5840 2022-11-23T01:09:00.7293942Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5841 2022-11-23T01:09:00.7294268Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5842 2022-11-23T01:09:00.7294614Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5843 2022-11-23T01:09:00.7295062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:09:00.7295395Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:09:00.7295822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:09:00.7296152Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:09:00.7296478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:09:00.7296969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:09:00.7297280Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:09:00.7297709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:09:00.7298055Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:09:00.7298376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:09:00.7298820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:09:00.7299149Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:09:00.7299575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:09:00.7299918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:09:00.7300225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:09:00.7300683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:09:00.7301010Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:09:00.7301423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:09:00.7301764Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:09:00.7302087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:09:00.7302449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:09:00.7302802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:09:00.7303169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:09:00.7303531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:09:00.7304012Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:09:00.7304528Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:09:00.7305045Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:09:00.7305558Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:09:00.7305830Z ok (1.820s) 2022-11-23T01:09:00.7306159Z test_prepare_output_error (__main__.TensorParallelStyleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5896 2022-11-23T01:09:00.7306561Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5897 2022-11-23T01:09:00.7306887Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5898 2022-11-23T01:09:00.7307227Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5899 2022-11-23T01:09:00.7307676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:09:00.7308007Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:09:00.7308422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:09:00.7308766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:09:00.7309092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:09:00.7309579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:09:00.7309892Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:09:00.7310320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:09:00.7310664Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:09:00.7310985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:09:00.7311429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:09:00.7311755Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:09:00.7312257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:09:00.7312587Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:09:00.7312910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:09:00.7313370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:09:00.7313700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:09:00.7314116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:09:00.7314460Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:09:00.7314781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:09:00.7315131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:09:00.7315497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:09:00.7315861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:09:00.7316220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:09:00.7316694Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:09:00.7317209Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:09:00.7317725Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:09:00.7318235Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:09:00.7318609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2022-11-23T01:09:00.7318975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T01:09:00.7319330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T01:09:00.7319856Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T01:09:00.7320235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-11-23T01:09:00.7320818Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T01:09:00.7321329Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T01:09:00.7321709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T01:09:00.7322076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2 2022-11-23T01:09:00.7322611Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T01:09:00.7323003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3 2022-11-23T01:09:00.7323352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T01:09:00.7323833Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes. 2022-11-23T01:09:00.7324341Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes. 2022-11-23T01:09:00.7324849Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes. 2022-11-23T01:09:00.7325220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2 2022-11-23T01:09:00.7325708Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes. 2022-11-23T01:09:00.7326096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3 2022-11-23T01:09:00.7326456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T01:09:00.7326801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T01:09:00.7327284Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes. 2022-11-23T01:09:00.7327793Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes. 2022-11-23T01:09:00.7328181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0 2022-11-23T01:09:00.7328652Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes. 2022-11-23T01:09:00.7329155Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes. 2022-11-23T01:09:00.7329543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1 2022-11-23T01:09:00.7329895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2 2022-11-23T01:09:00.7330251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3 2022-11-23T01:09:00.7330741Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes. 2022-11-23T01:09:00.7331245Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes. 2022-11-23T01:09:00.7331738Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes. 2022-11-23T01:09:00.7332245Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes. 2022-11-23T01:09:00.7332685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1 2022-11-23T01:09:00.7333045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 3 2022-11-23T01:09:00.7333391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 2 2022-11-23T01:09:00.7333751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0 2022-11-23T01:09:00.7334229Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:6 with 4 nodes. 2022-11-23T01:09:00.7334738Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 4 nodes. 2022-11-23T01:09:00.7335142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0 2022-11-23T01:09:00.7335629Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 4 nodes. 2022-11-23T01:09:00.7336138Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:6 with 4 nodes. 2022-11-23T01:09:00.7336528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 2 2022-11-23T01:09:00.7336875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1 2022-11-23T01:09:00.7337234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 3 2022-11-23T01:09:00.7337716Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 4 nodes. 2022-11-23T01:09:00.7338212Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:7 with 4 nodes. 2022-11-23T01:09:00.7338716Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:7 with 4 nodes. 2022-11-23T01:09:00.7339233Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 4 nodes. 2022-11-23T01:09:00.7339621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 2 2022-11-23T01:09:00.7339968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 3 2022-11-23T01:09:00.7340326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1 2022-11-23T01:09:00.7340680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0 2022-11-23T01:09:00.7341167Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 4 nodes. 2022-11-23T01:09:00.7341665Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 4 nodes. 2022-11-23T01:09:00.7342061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0 2022-11-23T01:09:00.7342545Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:8 with 4 nodes. 2022-11-23T01:09:00.7343059Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:8 with 4 nodes. 2022-11-23T01:09:00.7343433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1 2022-11-23T01:09:00.7343791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 3 2022-11-23T01:09:00.7344150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 2 2022-11-23T01:09:00.7344636Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:9 with 4 nodes. 2022-11-23T01:09:00.7345164Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:9 with 4 nodes. 2022-11-23T01:09:00.7345678Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 4 nodes. 2022-11-23T01:09:00.7346189Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 4 nodes. 2022-11-23T01:09:00.7346571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1 2022-11-23T01:09:00.7346942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 3 2022-11-23T01:09:00.7347312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0 2022-11-23T01:09:00.7347696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 2 2022-11-23T01:09:00.7348188Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 4 nodes. 2022-11-23T01:09:00.7348702Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:10 with 4 nodes. 2022-11-23T01:09:00.7349098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0 2022-11-23T01:09:00.7349451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 2 2022-11-23T01:09:00.7349941Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 4 nodes. 2022-11-23T01:09:00.7350447Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:10 with 4 nodes. 2022-11-23T01:09:00.7350827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1 2022-11-23T01:09:00.7351194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 3 2022-11-23T01:09:00.7351683Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 4 nodes. 2022-11-23T01:09:00.7352269Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 4 nodes. 2022-11-23T01:09:00.7352766Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:11 with 4 nodes. 2022-11-23T01:09:00.7353268Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:11 with 4 nodes. 2022-11-23T01:09:00.7353659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 2 2022-11-23T01:09:00.7354028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 3 2022-11-23T01:09:00.7354379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1 2022-11-23T01:09:00.7354746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0 2022-11-23T01:09:00.7355228Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 4 nodes. 2022-11-23T01:09:00.7355737Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 4 nodes. 2022-11-23T01:09:00.7356118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0 2022-11-23T01:09:00.7356597Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:12 with 4 nodes. 2022-11-23T01:09:00.7357104Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:12 with 4 nodes. 2022-11-23T01:09:00.7357495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1 2022-11-23T01:09:00.7357881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 3 2022-11-23T01:09:00.7358248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 2 2022-11-23T01:09:00.7358737Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 4 nodes. 2022-11-23T01:09:00.7359236Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:13 with 4 nodes. 2022-11-23T01:09:00.7359746Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:13 with 4 nodes. 2022-11-23T01:09:00.7360251Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 4 nodes. 2022-11-23T01:09:00.7360690Z ok (2.220s) 2022-11-23T01:09:00.7361017Z test_rowwise_parallel_style (__main__.TensorParallelStyleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6024 2022-11-23T01:09:00.7361416Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6025 2022-11-23T01:09:00.7361744Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6026 2022-11-23T01:09:00.7362064Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6027 2022-11-23T01:09:00.7362506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:09:00.7362844Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:09:00.7363268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:09:00.7363603Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:09:00.7363929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:09:00.7364389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:09:00.7364718Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:09:00.7365130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:09:00.7365474Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:09:00.7365797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:09:00.7366258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:09:00.7366571Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:09:00.7367006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:09:00.7367348Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:09:00.7367658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:09:00.7368113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:09:00.7368440Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:09:00.7368864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:09:00.7369195Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:09:00.7369516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:09:00.7369878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:09:00.7370335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:09:00.7370747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:09:00.7371108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:09:00.7371598Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:09:00.7372103Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:09:00.7372625Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:09:00.7373139Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:09:00.7373650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2022-11-23T01:09:00.7374001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T01:09:00.7374364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T01:09:00.7374720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-11-23T01:09:00.7375211Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T01:09:00.7375712Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T01:09:00.7376225Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T01:09:00.7376736Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T01:09:00.7377125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T01:09:00.7377474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2 2022-11-23T01:09:00.7377830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3 2022-11-23T01:09:00.7378194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T01:09:00.7378664Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes. 2022-11-23T01:09:00.7379174Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes. 2022-11-23T01:09:00.7379686Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes. 2022-11-23T01:09:00.7380074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2 2022-11-23T01:09:00.7380544Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes. 2022-11-23T01:09:00.7380929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3 2022-11-23T01:09:00.7381286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T01:09:00.7381645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T01:09:00.7382118Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes. 2022-11-23T01:09:00.7382623Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes. 2022-11-23T01:09:00.7383017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0 2022-11-23T01:09:00.7383417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1 2022-11-23T01:09:00.7383891Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes. 2022-11-23T01:09:00.7384404Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes. 2022-11-23T01:09:00.7384794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2 2022-11-23T01:09:00.7385143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3 2022-11-23T01:09:00.7385626Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes. 2022-11-23T01:09:00.7386168Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes. 2022-11-23T01:09:00.7386675Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes. 2022-11-23T01:09:00.7387168Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes. 2022-11-23T01:09:00.7387551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1 2022-11-23T01:09:00.7387910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 3 2022-11-23T01:09:00.7388269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0 2022-11-23T01:09:00.7388620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 2 2022-11-23T01:09:00.7389101Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 4 nodes. 2022-11-23T01:09:00.7389607Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:6 with 4 nodes. 2022-11-23T01:09:00.7389998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0 2022-11-23T01:09:00.7390346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 2 2022-11-23T01:09:00.7390829Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:6 with 4 nodes. 2022-11-23T01:09:00.7391340Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 4 nodes. 2022-11-23T01:09:00.7391728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 3 2022-11-23T01:09:00.7392157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1 2022-11-23T01:09:00.7392645Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:7 with 4 nodes. 2022-11-23T01:09:00.7393162Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 4 nodes. 2022-11-23T01:09:00.7393667Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 4 nodes. 2022-11-23T01:09:00.7394162Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:7 with 4 nodes. 2022-11-23T01:09:00.7394553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 2 2022-11-23T01:09:00.7394914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 3 2022-11-23T01:09:00.7395260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1 2022-11-23T01:09:00.7395619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0 2022-11-23T01:09:00.7396137Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 4 nodes. 2022-11-23T01:09:00.7396711Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 4 nodes. 2022-11-23T01:09:00.7397087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0 2022-11-23T01:09:00.7397565Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:8 with 4 nodes. 2022-11-23T01:09:00.7398070Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:8 with 4 nodes. 2022-11-23T01:09:00.7398459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1 2022-11-23T01:09:00.7398836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 3 2022-11-23T01:09:00.7399192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 2 2022-11-23T01:09:00.7399679Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:9 with 4 nodes. 2022-11-23T01:09:00.7400183Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:9 with 4 nodes. 2022-11-23T01:09:00.7400774Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 4 nodes. 2022-11-23T01:09:00.7401285Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 4 nodes. 2022-11-23T01:09:00.7401569Z ok (2.120s) 2022-11-23T01:09:00.7401674Z 2022-11-23T01:09:00.7401876Z ---------------------------------------------------------------------- 2022-11-23T01:09:00.7402105Z Ran 8 tests in 15.915s 2022-11-23T01:09:00.7402220Z 2022-11-23T01:09:00.7402280Z OK 2022-11-23T01:09:00.7402371Z 2022-11-23T01:09:00.7402455Z Generating XML reports... 2022-11-23T01:09:00.7402932Z Generated XML report: test-reports/python-unittest/distributed._tensor.parallel.test_tp_style/TEST-TensorParallelStyleTest-20221123010844.xml 2022-11-23T01:09:00.7403216Z 2022-11-23T01:09:00.7403563Z ##[endgroup] 2022-11-23T01:09:00.7404044Z FINISHED PRINTING LOG FILE of distributed/_tensor/parallel/test_tp_style (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-parallel-test_tp_style_joedliwc) 2022-11-23T01:09:00.7404320Z 2022-11-23T01:09:00.7404483Z Running distributed/test_store ... [2022-11-23 01:09:00.718343] 2022-11-23T01:09:00.7404993Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_store.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:09:00.718743] 2022-11-23T01:10:06.3906138Z 2022-11-23T01:10:06.3906878Z Expand the folded group to see the log file of distributed/test_store 2022-11-23T01:10:06.3907883Z ##[group]PRINTING LOG FILE of distributed/test_store (/var/lib/jenkins/workspace/test/test-reports/distributed-test_store_supfu_sg) 2022-11-23T01:10:06.3908856Z , <__main__.FileStoreTest testMethod=test_init_pg_and_rpc_with_same_file>, <__main__.FileStoreTest testMethod=test_refcount>, <__main__.FileStoreTest testMethod=test_set_get>]> 2022-11-23T01:10:06.3909434Z test_compare_set (__main__.FileStoreTest) 2022-11-23T01:10:06.3913810Z test_init_pg_and_rpc_with_same_file (__main__.FileStoreTest) 2022-11-23T01:10:06.3914198Z test_refcount (__main__.FileStoreTest) 2022-11-23T01:10:06.3914517Z test_set_get (__main__.FileStoreTest) 2022-11-23T01:10:06.3914928Z , <__main__.HashStoreTest testMethod=test_set_get>]> 2022-11-23T01:10:06.3915305Z test_compare_set (__main__.HashStoreTest) 2022-11-23T01:10:06.3915514Z test_set_get (__main__.HashStoreTest) 2022-11-23T01:10:06.3916045Z , <__main__.PrefixFileStoreTest testMethod=test_set_get>]> 2022-11-23T01:10:06.3916419Z test_compare_set (__main__.PrefixFileStoreTest) 2022-11-23T01:10:06.3916652Z test_set_get (__main__.PrefixFileStoreTest) 2022-11-23T01:10:06.3920373Z ]> 2022-11-23T01:10:06.3921114Z test_get_underlying_store (__main__.PrefixStoreTest) 2022-11-23T01:10:06.3921847Z , <__main__.PrefixTCPStoreTest testMethod=test_set_get>]> 2022-11-23T01:10:06.3922504Z test_compare_set (__main__.PrefixTCPStoreTest) 2022-11-23T01:10:06.3922921Z test_set_get (__main__.PrefixTCPStoreTest) 2022-11-23T01:10:06.3923355Z ]> 2022-11-23T01:10:06.3923642Z test_set_get (__main__.PythonStoreTest) 2022-11-23T01:10:06.3923935Z ]> 2022-11-23T01:10:06.3924216Z test_nominal (__main__.RendezvousEnvTest) 2022-11-23T01:10:06.3924581Z , <__main__.RendezvousFileTest testMethod=test_nominal>]> 2022-11-23T01:10:06.3924944Z test_common_errors (__main__.RendezvousFileTest) 2022-11-23T01:10:06.3961270Z test_nominal (__main__.RendezvousFileTest) 2022-11-23T01:10:06.3962091Z , <__main__.RendezvousTCPTest testMethod=test_dns_timeout>, <__main__.RendezvousTCPTest testMethod=test_nominal>, <__main__.RendezvousTCPTest testMethod=test_tcp_store_timeout_set>]> 2022-11-23T01:10:06.3963070Z test_common_errors (__main__.RendezvousTCPTest) 2022-11-23T01:10:06.3963517Z test_dns_timeout (__main__.RendezvousTCPTest) 2022-11-23T01:10:06.3963945Z test_nominal (__main__.RendezvousTCPTest) 2022-11-23T01:10:06.3964344Z test_tcp_store_timeout_set (__main__.RendezvousTCPTest) 2022-11-23T01:10:06.3964966Z , <__main__.RendezvousTest testMethod=test_url_with_node_params>]> 2022-11-23T01:10:06.3965585Z test_unknown_handler (__main__.RendezvousTest) 2022-11-23T01:10:06.3966036Z test_url_with_node_params (__main__.RendezvousTest) 2022-11-23T01:10:06.3967271Z , <__main__.TCPStoreTest testMethod=test_compare_set>, <__main__.TCPStoreTest testMethod=test_init_pg_and_rpc_with_same_socket>, <__main__.TCPStoreTest testMethod=test_multi_worker_with_fixed_world_size>, <__main__.TCPStoreTest testMethod=test_multi_worker_with_nonfixed_world_size>, <__main__.TCPStoreTest testMethod=test_multitenancy>, <__main__.TCPStoreTest testMethod=test_numkeys_delkeys>, <__main__.TCPStoreTest testMethod=test_set_get>]> 2022-11-23T01:10:06.3968463Z test_address_already_in_use (__main__.TCPStoreTest) 2022-11-23T01:10:06.3968886Z test_compare_set (__main__.TCPStoreTest) 2022-11-23T01:10:06.3969282Z test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest) 2022-11-23T01:10:06.3969731Z test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest) 2022-11-23T01:10:06.3970227Z test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest) 2022-11-23T01:10:06.3970683Z test_multitenancy (__main__.TCPStoreTest) 2022-11-23T01:10:06.3971101Z test_numkeys_delkeys (__main__.TCPStoreTest) 2022-11-23T01:10:06.3971502Z test_set_get (__main__.TCPStoreTest) 2022-11-23T01:10:06.3972253Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T01:10:06.3973066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:06.3973681Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:06.3974196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:06.3974708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:06.3974905Z 2022-11-23T01:10:06.3975025Z Running tests... 2022-11-23T01:10:06.3975524Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.3976026Z test_compare_set (__main__.FileStoreTest) ... ok (0.264s) 2022-11-23T01:10:06.3976276Z 2022-11-23T01:10:06.3976624Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.3977054Z Ran 1 test in 0.264s 2022-11-23T01:10:06.3977252Z 2022-11-23T01:10:06.3977359Z OK 2022-11-23T01:10:06.3977524Z 2022-11-23T01:10:06.3977654Z Generating XML reports... 2022-11-23T01:10:06.3978519Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123010903.xml 2022-11-23T01:10:06.3979081Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T01:10:06.3979601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:06.3980109Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:06.3980831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:06.3993948Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:06.3994128Z 2022-11-23T01:10:06.3994204Z Running tests... 2022-11-23T01:10:06.3994546Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.3995019Z test_init_pg_and_rpc_with_same_file (__main__.FileStoreTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:10:06.3995599Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T01:10:06.3995961Z ok (0.305s) 2022-11-23T01:10:06.3996115Z 2022-11-23T01:10:06.3996472Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.3996907Z Ran 1 test in 0.306s 2022-11-23T01:10:06.3997107Z 2022-11-23T01:10:06.3997203Z OK 2022-11-23T01:10:06.3997331Z 2022-11-23T01:10:06.3997458Z Generating XML reports... 2022-11-23T01:10:06.3998145Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123010904.xml 2022-11-23T01:10:06.3998897Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T01:10:06.3999618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:06.4000138Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:06.4001009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:06.4001589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:06.4001869Z 2022-11-23T01:10:06.4001986Z Running tests... 2022-11-23T01:10:06.4002542Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4003013Z test_refcount (__main__.FileStoreTest) ... ok (0.264s) 2022-11-23T01:10:06.4003231Z 2022-11-23T01:10:06.4003548Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4003982Z Ran 1 test in 0.264s 2022-11-23T01:10:06.4004175Z 2022-11-23T01:10:06.4004274Z OK 2022-11-23T01:10:06.4004438Z 2022-11-23T01:10:06.4004568Z Generating XML reports... 2022-11-23T01:10:06.4005264Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123010906.xml 2022-11-23T01:10:06.4006061Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T01:10:06.4006813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:06.4007513Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:06.4008253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:06.4008878Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:06.4009174Z 2022-11-23T01:10:06.4009289Z Running tests... 2022-11-23T01:10:06.4009824Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4010319Z test_set_get (__main__.FileStoreTest) ... ok (0.263s) 2022-11-23T01:10:06.4010583Z 2022-11-23T01:10:06.4010935Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4011403Z Ran 1 test in 0.263s 2022-11-23T01:10:06.4011602Z 2022-11-23T01:10:06.4011712Z OK 2022-11-23T01:10:06.4011875Z 2022-11-23T01:10:06.4012025Z Generating XML reports... 2022-11-23T01:10:06.4012650Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123010908.xml 2022-11-23T01:10:06.4013100Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T01:10:06.4013559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:06.4013894Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:06.4014310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:06.4014656Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:06.4014822Z 2022-11-23T01:10:06.4014895Z Running tests... 2022-11-23T01:10:06.4015197Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4015470Z test_compare_set (__main__.HashStoreTest) ... ok (0.264s) 2022-11-23T01:10:06.4015618Z 2022-11-23T01:10:06.4015810Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4016049Z Ran 1 test in 0.264s 2022-11-23T01:10:06.4016163Z 2022-11-23T01:10:06.4016212Z OK 2022-11-23T01:10:06.4016304Z 2022-11-23T01:10:06.4016390Z Generating XML reports... 2022-11-23T01:10:06.4016803Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20221123010910.xml 2022-11-23T01:10:06.4017255Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T01:10:06.4017701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:06.4018031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:06.4018459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:06.4018811Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:06.4018962Z 2022-11-23T01:10:06.4019034Z Running tests... 2022-11-23T01:10:06.4019335Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4019615Z test_set_get (__main__.HashStoreTest) ... ok (0.263s) 2022-11-23T01:10:06.4019761Z 2022-11-23T01:10:06.4019937Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4020173Z Ran 1 test in 0.263s 2022-11-23T01:10:06.4020284Z 2022-11-23T01:10:06.4020347Z OK 2022-11-23T01:10:06.4020439Z 2022-11-23T01:10:06.4020510Z Generating XML reports... 2022-11-23T01:10:06.4020920Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20221123010912.xml 2022-11-23T01:10:06.4021370Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T01:10:06.4021996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:06.4022315Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:06.4022790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:06.4023137Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:06.4023299Z 2022-11-23T01:10:06.4023370Z Running tests... 2022-11-23T01:10:06.4023656Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4023953Z test_compare_set (__main__.PrefixFileStoreTest) ... ok (0.263s) 2022-11-23T01:10:06.4024113Z 2022-11-23T01:10:06.4024306Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4024528Z Ran 1 test in 0.263s 2022-11-23T01:10:06.4024667Z 2022-11-23T01:10:06.4024826Z OK 2022-11-23T01:10:06.4024942Z 2022-11-23T01:10:06.4025029Z Generating XML reports... 2022-11-23T01:10:06.4025467Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20221123010914.xml 2022-11-23T01:10:06.4025925Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T01:10:06.4026387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:06.4026718Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:06.4027131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:06.4027478Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:06.4027643Z 2022-11-23T01:10:06.4027716Z Running tests... 2022-11-23T01:10:06.4028012Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4028290Z test_set_get (__main__.PrefixFileStoreTest) ... ok (0.264s) 2022-11-23T01:10:06.4028445Z 2022-11-23T01:10:06.4028638Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4028874Z Ran 1 test in 0.264s 2022-11-23T01:10:06.4028986Z 2022-11-23T01:10:06.4029036Z OK 2022-11-23T01:10:06.4029127Z 2022-11-23T01:10:06.4029212Z Generating XML reports... 2022-11-23T01:10:06.4029639Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20221123010915.xml 2022-11-23T01:10:06.4030108Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T01:10:06.4030556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:06.4030891Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:06.4031320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:06.4031658Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:06.4031825Z 2022-11-23T01:10:06.4031975Z Running tests... 2022-11-23T01:10:06.4032281Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4032586Z test_get_underlying_store (__main__.PrefixStoreTest) ... ok (0.002s) 2022-11-23T01:10:06.4032733Z 2022-11-23T01:10:06.4032929Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4033169Z Ran 1 test in 0.003s 2022-11-23T01:10:06.4033282Z 2022-11-23T01:10:06.4033344Z OK 2022-11-23T01:10:06.4033435Z 2022-11-23T01:10:06.4033505Z Generating XML reports... 2022-11-23T01:10:06.4033930Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixStoreTest-20221123010917.xml 2022-11-23T01:10:06.4034389Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T01:10:06.4034850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:06.4035170Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:06.4035746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:06.4036375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:06.4036628Z 2022-11-23T01:10:06.4036741Z Running tests... 2022-11-23T01:10:06.4037314Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4037652Z test_compare_set (__main__.PrefixTCPStoreTest) ... ok (0.268s) 2022-11-23T01:10:06.4037815Z 2022-11-23T01:10:06.4038012Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4038240Z Ran 1 test in 0.268s 2022-11-23T01:10:06.4038352Z 2022-11-23T01:10:06.4038415Z OK 2022-11-23T01:10:06.4038507Z 2022-11-23T01:10:06.4038591Z Generating XML reports... 2022-11-23T01:10:06.4039058Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20221123010919.xml 2022-11-23T01:10:06.4039525Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T01:10:06.4039985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:06.4040317Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:06.4040873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:06.4041228Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:06.4041391Z 2022-11-23T01:10:06.4041466Z Running tests... 2022-11-23T01:10:06.4041759Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4042052Z test_set_get (__main__.PrefixTCPStoreTest) ... ok (0.267s) 2022-11-23T01:10:06.4042208Z 2022-11-23T01:10:06.4042404Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4042643Z Ran 1 test in 0.267s 2022-11-23T01:10:06.4042741Z 2022-11-23T01:10:06.4042802Z OK 2022-11-23T01:10:06.4042893Z 2022-11-23T01:10:06.4042978Z Generating XML reports... 2022-11-23T01:10:06.4043411Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20221123010920.xml 2022-11-23T01:10:06.4043864Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T01:10:06.4044321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:06.4044659Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:06.4045088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:06.4045420Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:06.4045586Z 2022-11-23T01:10:06.4045660Z Running tests... 2022-11-23T01:10:06.4045962Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4046232Z test_set_get (__main__.PythonStoreTest) ... ok (0.263s) 2022-11-23T01:10:06.4046380Z 2022-11-23T01:10:06.4046575Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4046809Z Ran 1 test in 0.263s 2022-11-23T01:10:06.4046922Z 2022-11-23T01:10:06.4046984Z OK 2022-11-23T01:10:06.4047062Z 2022-11-23T01:10:06.4047145Z Generating XML reports... 2022-11-23T01:10:06.4047567Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20221123010922.xml 2022-11-23T01:10:06.4048028Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T01:10:06.4048469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:06.4048803Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:06.4049233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:06.4049649Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:06.4049819Z 2022-11-23T01:10:06.4049879Z Running tests... 2022-11-23T01:10:06.4050185Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4050476Z test_nominal (__main__.RendezvousEnvTest) ... ok (0.262s) 2022-11-23T01:10:06.4050629Z 2022-11-23T01:10:06.4050807Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4051043Z Ran 1 test in 0.263s 2022-11-23T01:10:06.4051155Z 2022-11-23T01:10:06.4051217Z OK 2022-11-23T01:10:06.4051309Z 2022-11-23T01:10:06.4051394Z Generating XML reports... 2022-11-23T01:10:06.4051808Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20221123010924.xml 2022-11-23T01:10:06.4052316Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T01:10:06.4052774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:06.4053095Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:06.4053523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:06.4053870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:06.4054037Z 2022-11-23T01:10:06.4054110Z Running tests... 2022-11-23T01:10:06.4054399Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4054694Z test_common_errors (__main__.RendezvousFileTest) ... ok (0.261s) 2022-11-23T01:10:06.4054854Z 2022-11-23T01:10:06.4055046Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4055272Z Ran 1 test in 0.261s 2022-11-23T01:10:06.4055386Z 2022-11-23T01:10:06.4055447Z OK 2022-11-23T01:10:06.4055538Z 2022-11-23T01:10:06.4055620Z Generating XML reports... 2022-11-23T01:10:06.4056050Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20221123010926.xml 2022-11-23T01:10:06.4056505Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T01:10:06.4056959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:06.4057287Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:06.4057702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:06.4058049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:06.4058215Z 2022-11-23T01:10:06.4058290Z Running tests... 2022-11-23T01:10:06.4058590Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4058868Z test_nominal (__main__.RendezvousFileTest) ... ok (0.266s) 2022-11-23T01:10:06.4059018Z 2022-11-23T01:10:06.4059215Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4059452Z Ran 1 test in 0.266s 2022-11-23T01:10:06.4059564Z 2022-11-23T01:10:06.4059612Z OK 2022-11-23T01:10:06.4059703Z 2022-11-23T01:10:06.4059786Z Generating XML reports... 2022-11-23T01:10:06.4060215Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20221123010928.xml 2022-11-23T01:10:06.4060682Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T01:10:06.4061126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:06.4061456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:06.4061886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:06.4062234Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:06.4062385Z 2022-11-23T01:10:06.4062496Z Running tests... 2022-11-23T01:10:06.4062800Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4063094Z test_common_errors (__main__.RendezvousTCPTest) ... ok (0.262s) 2022-11-23T01:10:06.4063252Z 2022-11-23T01:10:06.4063433Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4063669Z Ran 1 test in 0.262s 2022-11-23T01:10:06.4063781Z 2022-11-23T01:10:06.4063844Z OK 2022-11-23T01:10:06.4063934Z 2022-11-23T01:10:06.4064004Z Generating XML reports... 2022-11-23T01:10:06.4064428Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123010929.xml 2022-11-23T01:10:06.4064920Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T01:10:06.4065378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:06.4065699Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:06.4066128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:06.4066478Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:06.4066641Z 2022-11-23T01:10:06.4066714Z Running tests... 2022-11-23T01:10:06.4067001Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4067555Z test_dns_timeout (__main__.RendezvousTCPTest) ... [W socket.cpp:601] [c10d] The IPv6 network addresses of (dnsnotexist, 23456) cannot be retrieved (gai error: -2 - Name or service not known). 2022-11-23T01:10:06.4068005Z [E socket.cpp:860] [c10d] The client socket has timed out after 1s while trying to connect to (dnsnotexist, 23456). 2022-11-23T01:10:06.4068255Z ok (0.270s) 2022-11-23T01:10:06.4068359Z 2022-11-23T01:10:06.4068555Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4068796Z Ran 1 test in 0.270s 2022-11-23T01:10:06.4068910Z 2022-11-23T01:10:06.4068974Z OK 2022-11-23T01:10:06.4069051Z 2022-11-23T01:10:06.4069135Z Generating XML reports... 2022-11-23T01:10:06.4069562Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123010931.xml 2022-11-23T01:10:06.4070029Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T01:10:06.4070473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:06.4070809Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:06.4071236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:06.4071592Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:06.4071742Z 2022-11-23T01:10:06.4071816Z Running tests... 2022-11-23T01:10:06.4072204Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4072495Z test_nominal (__main__.RendezvousTCPTest) ... ok (0.264s) 2022-11-23T01:10:06.4072646Z 2022-11-23T01:10:06.4072825Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4073066Z Ran 1 test in 0.264s 2022-11-23T01:10:06.4073179Z 2022-11-23T01:10:06.4073240Z OK 2022-11-23T01:10:06.4073332Z 2022-11-23T01:10:06.4073418Z Generating XML reports... 2022-11-23T01:10:06.4073831Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123010933.xml 2022-11-23T01:10:06.4074301Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T01:10:06.4074763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:06.4075083Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:06.4075560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:06.4075909Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:06.4076073Z 2022-11-23T01:10:06.4076146Z Running tests... 2022-11-23T01:10:06.4076433Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4076741Z test_tcp_store_timeout_set (__main__.RendezvousTCPTest) ... ok (10.343s) 2022-11-23T01:10:06.4076911Z 2022-11-23T01:10:06.4077107Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4077330Z Ran 1 test in 10.343s 2022-11-23T01:10:06.4077445Z 2022-11-23T01:10:06.4077540Z OK 2022-11-23T01:10:06.4077631Z 2022-11-23T01:10:06.4077714Z Generating XML reports... 2022-11-23T01:10:06.4078142Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123010935.xml 2022-11-23T01:10:06.4078593Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T01:10:06.4079050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:06.4079384Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:06.4079799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:06.4080148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:06.4080313Z 2022-11-23T01:10:06.4080387Z Running tests... 2022-11-23T01:10:06.4080857Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4081148Z test_unknown_handler (__main__.RendezvousTest) ... ok (0.262s) 2022-11-23T01:10:06.4081305Z 2022-11-23T01:10:06.4081503Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4081739Z Ran 1 test in 0.262s 2022-11-23T01:10:06.4081857Z 2022-11-23T01:10:06.4081905Z OK 2022-11-23T01:10:06.4081998Z 2022-11-23T01:10:06.4082082Z Generating XML reports... 2022-11-23T01:10:06.4082503Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20221123010947.xml 2022-11-23T01:10:06.4082964Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T01:10:06.4083410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:06.4083742Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:06.4084174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:06.4084511Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:06.4084678Z 2022-11-23T01:10:06.4084751Z Running tests... 2022-11-23T01:10:06.4085051Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4085351Z test_url_with_node_params (__main__.RendezvousTest) ... ok (0.261s) 2022-11-23T01:10:06.4085498Z 2022-11-23T01:10:06.4085690Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4085931Z Ran 1 test in 0.262s 2022-11-23T01:10:06.4086043Z 2022-11-23T01:10:06.4086105Z OK 2022-11-23T01:10:06.4086199Z 2022-11-23T01:10:06.4086269Z Generating XML reports... 2022-11-23T01:10:06.4086689Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20221123010949.xml 2022-11-23T01:10:06.4087145Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T01:10:06.4087605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:06.4087921Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:06.4088422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:06.4088773Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:06.4088938Z 2022-11-23T01:10:06.4089011Z Running tests... 2022-11-23T01:10:06.4089300Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4089846Z test_address_already_in_use (__main__.TCPStoreTest) ... [W socket.cpp:426] [c10d] The server socket has failed to bind to [::]:33145 (errno: 98 - Address already in use). 2022-11-23T01:10:06.4090352Z [W socket.cpp:426] [c10d] The server socket has failed to bind to 0.0.0.0:33145 (errno: 98 - Address already in use). 2022-11-23T01:10:06.4090684Z [E socket.cpp:462] [c10d] The server socket has failed to listen on any local network address. 2022-11-23T01:10:06.4090972Z ok (0.264s) 2022-11-23T01:10:06.4091073Z 2022-11-23T01:10:06.4091269Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4091513Z Ran 1 test in 0.264s 2022-11-23T01:10:06.4091612Z 2022-11-23T01:10:06.4091678Z OK 2022-11-23T01:10:06.4091768Z 2022-11-23T01:10:06.4091852Z Generating XML reports... 2022-11-23T01:10:06.4092264Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123010950.xml 2022-11-23T01:10:06.4092701Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T01:10:06.4093162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:06.4093493Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:06.4093922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:06.4094262Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:06.4094429Z 2022-11-23T01:10:06.4094502Z Running tests... 2022-11-23T01:10:06.4094806Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4095080Z test_compare_set (__main__.TCPStoreTest) ... ok (0.268s) 2022-11-23T01:10:06.4095228Z 2022-11-23T01:10:06.4095424Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4095662Z Ran 1 test in 0.268s 2022-11-23T01:10:06.4095775Z 2022-11-23T01:10:06.4095836Z OK 2022-11-23T01:10:06.4095913Z 2022-11-23T01:10:06.4095997Z Generating XML reports... 2022-11-23T01:10:06.4096411Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123010952.xml 2022-11-23T01:10:06.4096863Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T01:10:06.4097310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:06.4097638Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:06.4098072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:06.4098418Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:06.4098581Z 2022-11-23T01:10:06.4098643Z Running tests... 2022-11-23T01:10:06.4098941Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4099352Z test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:10:06.4099911Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T01:10:06.4100196Z ok (0.329s) 2022-11-23T01:10:06.4100298Z 2022-11-23T01:10:06.4100494Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4100733Z Ran 1 test in 0.329s 2022-11-23T01:10:06.4100844Z 2022-11-23T01:10:06.4100894Z OK 2022-11-23T01:10:06.4100986Z 2022-11-23T01:10:06.4101070Z Generating XML reports... 2022-11-23T01:10:06.4101528Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123010954.xml 2022-11-23T01:10:06.4101970Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T01:10:06.4102429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:06.4102768Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:06.4103201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:06.4103534Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:06.4103736Z 2022-11-23T01:10:06.4103809Z Running tests... 2022-11-23T01:10:06.4104113Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4104428Z test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest) ... ok (0.264s) 2022-11-23T01:10:06.4104590Z 2022-11-23T01:10:06.4104784Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4105025Z Ran 1 test in 0.264s 2022-11-23T01:10:06.4105136Z 2022-11-23T01:10:06.4105199Z OK 2022-11-23T01:10:06.4105288Z 2022-11-23T01:10:06.4105358Z Generating XML reports... 2022-11-23T01:10:06.4105773Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123010956.xml 2022-11-23T01:10:06.4106224Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T01:10:06.4106687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:06.4107010Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:06.4107438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:06.4107787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:06.4107950Z 2022-11-23T01:10:06.4108009Z Running tests... 2022-11-23T01:10:06.4108314Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4108628Z test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest) ... ok (0.263s) 2022-11-23T01:10:06.4108807Z 2022-11-23T01:10:06.4109000Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4109225Z Ran 1 test in 0.264s 2022-11-23T01:10:06.4109337Z 2022-11-23T01:10:06.4109400Z OK 2022-11-23T01:10:06.4109492Z 2022-11-23T01:10:06.4109576Z Generating XML reports... 2022-11-23T01:10:06.4109978Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123010958.xml 2022-11-23T01:10:06.4110430Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T01:10:06.4110885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:06.4111216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:06.4111630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:06.4112056Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:06.4112228Z 2022-11-23T01:10:06.4112303Z Running tests... 2022-11-23T01:10:06.4112590Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4112879Z test_multitenancy (__main__.TCPStoreTest) ... ok (0.266s) 2022-11-23T01:10:06.4113037Z 2022-11-23T01:10:06.4113238Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4113480Z Ran 1 test in 0.266s 2022-11-23T01:10:06.4113579Z 2022-11-23T01:10:06.4113642Z OK 2022-11-23T01:10:06.4113738Z 2022-11-23T01:10:06.4113822Z Generating XML reports... 2022-11-23T01:10:06.4114284Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123010959.xml 2022-11-23T01:10:06.4114727Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T01:10:06.4115187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:06.4115516Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:06.4115947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:06.4116278Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:06.4116499Z 2022-11-23T01:10:06.4116573Z Running tests... 2022-11-23T01:10:06.4116876Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4117155Z test_numkeys_delkeys (__main__.TCPStoreTest) ... ok (2.289s) 2022-11-23T01:10:06.4117307Z 2022-11-23T01:10:06.4117503Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4117742Z Ran 1 test in 2.289s 2022-11-23T01:10:06.4117857Z 2022-11-23T01:10:06.4117919Z OK 2022-11-23T01:10:06.4117995Z 2022-11-23T01:10:06.4118080Z Generating XML reports... 2022-11-23T01:10:06.4118497Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123011001.xml 2022-11-23T01:10:06.4118948Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T01:10:06.4119394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:06.4119729Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:06.4120156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:06.4120502Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:06.4120805Z 2022-11-23T01:10:06.4120884Z Running tests... 2022-11-23T01:10:06.4121197Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4121485Z test_set_get (__main__.TCPStoreTest) ... ok (0.262s) 2022-11-23T01:10:06.4121630Z 2022-11-23T01:10:06.4121808Z ---------------------------------------------------------------------- 2022-11-23T01:10:06.4122049Z Ran 1 test in 0.262s 2022-11-23T01:10:06.4122163Z 2022-11-23T01:10:06.4122226Z OK 2022-11-23T01:10:06.4122319Z 2022-11-23T01:10:06.4122403Z Generating XML reports... 2022-11-23T01:10:06.4122803Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123011005.xml 2022-11-23T01:10:06.4123040Z 2022-11-23T01:10:06.4123410Z ##[endgroup] 2022-11-23T01:10:06.4123825Z FINISHED PRINTING LOG FILE of distributed/test_store (/var/lib/jenkins/workspace/test/test-reports/distributed-test_store_supfu_sg) 2022-11-23T01:10:06.4124061Z 2022-11-23T01:10:06.4124267Z Running distributed/elastic/multiprocessing/api_test ... [2022-11-23 01:10:06.391467] 2022-11-23T01:10:06.4124826Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/multiprocessing/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:10:06.391816] 2022-11-23T01:10:27.8836554Z 2022-11-23T01:10:27.8837131Z Expand the folded group to see the log file of distributed/elastic/multiprocessing/api_test 2022-11-23T01:10:27.8838340Z ##[group]PRINTING LOG FILE of distributed/elastic/multiprocessing/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-multiprocessing-api_test_6z66dnfd) 2022-11-23T01:10:27.8842327Z Test results will be stored in test-reports/python-unittest/distributed.elastic.multiprocessing.api_test 2022-11-23T01:10:27.8842906Z 2022-11-23T01:10:27.8843023Z Running tests... 2022-11-23T01:10:27.8843517Z ---------------------------------------------------------------------- 2022-11-23T01:10:27.8844018Z test_get_failures (__main__.RunProcResultsTest) ... ok (0.266s) 2022-11-23T01:10:27.8844713Z test_is_failed (__main__.RunProcResultsTest) ... ok (0.001s) 2022-11-23T01:10:27.8845202Z test_args_env_len_mismatch (__main__.StartProcessesListTest) ... ok (0.001s) 2022-11-23T01:10:27.8845729Z test_binary (__main__.StartProcessesListTest) ... hello stderr from 0 2022-11-23T01:10:27.8846123Z hello stdout from 0 2022-11-23T01:10:27.8846420Z hello stderr from 1 2022-11-23T01:10:27.8846742Z hello stdout from 1 2022-11-23T01:10:27.8847032Z ok (0.120s) 2022-11-23T01:10:27.8847467Z test_binary_exit (__main__.StartProcessesListTest) ... bar stderr from 1 2022-11-23T01:10:27.8847945Z bar stdout from 1 2022-11-23T01:10:27.8848437Z failed (exitcode: 138) local_rank: 0 (pid: 6565) of binary: distributed/elastic/multiprocessing/bin/echo1.py 2022-11-23T01:10:27.8849037Z ok (0.115s) 2022-11-23T01:10:27.8849438Z test_binary_incorrect_entrypoint (__main__.StartProcessesListTest) ... ok (0.009s) 2022-11-23T01:10:27.8850042Z test_binary_raises (__main__.StartProcessesListTest) ... Traceback (most recent call last): 2022-11-23T01:10:27.8850629Z File "distributed/elastic/multiprocessing/bin/echo2.py", line 22, in 2022-11-23T01:10:27.8851119Z raise RuntimeError(f"raised from {rank}") 2022-11-23T01:10:27.8851502Z RuntimeError: raised from 0 2022-11-23T01:10:27.8851795Z bar from 1 2022-11-23T01:10:27.8852273Z failed (exitcode: 1) local_rank: 0 (pid: 6568) of binary: distributed/elastic/multiprocessing/bin/echo2.py 2022-11-23T01:10:27.8852729Z ok (0.113s) 2022-11-23T01:10:27.8853153Z test_binary_redirect_and_tee (__main__.StartProcessesListTest) ... world stdout from 1 2022-11-23T01:10:27.8853613Z [trainer1]:world stderr from 1 2022-11-23T01:10:27.8853919Z ok (1.117s) 2022-11-23T01:10:27.8854350Z test_function (__main__.StartProcessesListTest) ... [trainer0]:hello stdout from 0 2022-11-23T01:10:27.8855201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:27.8855734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:27.8856537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:27.8857149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:27.8857960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:27.8858573Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:27.8859329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:27.8859941Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:27.8860343Z hello stdout from 1 2022-11-23T01:10:27.8860619Z hello stderr from 1 2022-11-23T01:10:27.8860859Z hello stdout from 0 2022-11-23T01:10:27.8861130Z hello stderr from 0 2022-11-23T01:10:27.8861443Z Closing process 6575 via signal SIGTERM 2022-11-23T01:10:27.8861729Z ok (1.675s) 2022-11-23T01:10:27.8862509Z test_function_large_ret_val (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:27.8863119Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:27.8863802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:27.8864365Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:27.8865069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:27.8865597Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:27.8866265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:27.8866919Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:27.8867584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:27.8868147Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:27.8868860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:27.8869423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:27.8870132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:27.8870784Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:27.8871578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:27.8872301Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:27.8872748Z Closing process 6595 via signal SIGTERM 2022-11-23T01:10:27.8873068Z ok (2.090s) 2022-11-23T01:10:27.8873450Z test_function_raise (__main__.StartProcessesListTest) 2022-11-23T01:10:27.8874385Z run 2x copies of echo2, raise an exception on the first ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:27.8875046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:27.8875833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:27.8876419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:27.8877120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:27.8877646Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:27.8878354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:27.8878906Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:27.8879360Z failed (exitcode: 1) local_rank: 0 (pid: 6635) of fn: echo2 (start_method: spawn) 2022-11-23T01:10:27.8879766Z Traceback (most recent call last): 2022-11-23T01:10:27.8880424Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll 2022-11-23T01:10:27.8881110Z self._pc.join(-1) 2022-11-23T01:10:27.8881625Z File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 160, in join 2022-11-23T01:10:27.8882161Z raise ProcessRaisedException(msg, error_index, failed_process.pid) 2022-11-23T01:10:27.8882651Z torch.multiprocessing.spawn.ProcessRaisedException: 2022-11-23T01:10:27.8882959Z 2022-11-23T01:10:27.8883215Z -- Process 0 terminated with the following error: 2022-11-23T01:10:27.8883569Z Traceback (most recent call last): 2022-11-23T01:10:27.8884126Z File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap 2022-11-23T01:10:27.8884518Z fn(i, *args) 2022-11-23T01:10:27.8885112Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 371, in _wrap 2022-11-23T01:10:27.8885544Z ret = record(fn)(*args_) 2022-11-23T01:10:27.8886170Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper 2022-11-23T01:10:27.8886644Z return f(*args, **kwargs) 2022-11-23T01:10:27.8887070Z File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2 2022-11-23T01:10:27.8887536Z raise RuntimeError(msg) 2022-11-23T01:10:27.8887804Z RuntimeError: hello 2022-11-23T01:10:27.8887969Z 2022-11-23T01:10:27.8888069Z ok (1.760s) 2022-11-23T01:10:27.8888597Z test_function_with_tensor (__main__.StartProcessesListTest) ... ok (0.002s) 2022-11-23T01:10:27.8889045Z test_invalid_log_dir (__main__.StartProcessesListTest) ... ok (0.001s) 2022-11-23T01:10:27.8889474Z test_multiprocess_context_close (__main__.StartProcessesListTest) ... Closing process 6655 via signal SIGTERM 2022-11-23T01:10:27.8889746Z ok (0.009s) 2022-11-23T01:10:27.8890223Z test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesListTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn) 2022-11-23T01:10:27.8890562Z Traceback (most recent call last): 2022-11-23T01:10:27.8890963Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll 2022-11-23T01:10:27.8891362Z self._pc.join(-1) 2022-11-23T01:10:27.8891603Z File "/opt/conda/lib/python3.7/unittest/mock.py", line 1016, in __call__ 2022-11-23T01:10:27.8891851Z return _mock_self._mock_call(*args, **kwargs) 2022-11-23T01:10:27.8892122Z File "/opt/conda/lib/python3.7/unittest/mock.py", line 1076, in _mock_call 2022-11-23T01:10:27.8892352Z raise effect 2022-11-23T01:10:27.8892613Z torch.multiprocessing.spawn.ProcessRaisedException: test msg 2022-11-23T01:10:27.8892875Z ok (0.005s) 2022-11-23T01:10:27.8893347Z test_pcontext_wait (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:27.8893724Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:27.8894156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:27.8894508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:27.8894734Z ok (2.648s) 2022-11-23T01:10:27.8895008Z test_subprocess_context_close (__main__.StartProcessesListTest) ... Sending process 6666 closing signal SIGTERM 2022-11-23T01:10:27.8895276Z ok (0.010s) 2022-11-23T01:10:27.8895501Z test_to_map (__main__.StartProcessesListTest) ... ok (0.002s) 2022-11-23T01:10:27.8895780Z test_validate_full_rank (__main__.StartProcessesListTest) ... ok (0.001s) 2022-11-23T01:10:27.8896315Z test_void_function (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:27.8896699Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:27.8897125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:27.8897460Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:27.8897893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:27.8898222Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:27.8898634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:27.8898974Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:27.8899192Z hello 2022-11-23T01:10:27.8899339Z world 2022-11-23T01:10:27.8899508Z Closing process 6668 via signal SIGTERM 2022-11-23T01:10:27.8899699Z ok (1.667s) 2022-11-23T01:10:27.8899935Z test_args_env_len_mismatch (__main__.StartProcessesTest) ... ok (0.002s) 2022-11-23T01:10:27.8900218Z test_binary_exit (__main__.StartProcessesTest) ... bar stderr from 1 2022-11-23T01:10:27.8900445Z bar stdout from 1 2022-11-23T01:10:27.8900731Z failed (exitcode: 138) local_rank: 0 (pid: 6687) of binary: distributed/elastic/multiprocessing/bin/echo1.py 2022-11-23T01:10:27.8900984Z ok (0.115s) 2022-11-23T01:10:27.8901225Z test_binary_incorrect_entrypoint (__main__.StartProcessesTest) ... ok (0.009s) 2022-11-23T01:10:27.8901550Z test_binary_raises (__main__.StartProcessesTest) ... Traceback (most recent call last): 2022-11-23T01:10:27.8901911Z File "distributed/elastic/multiprocessing/bin/echo2.py", line 22, in 2022-11-23T01:10:27.8902187Z raise RuntimeError(f"raised from {rank}") 2022-11-23T01:10:27.8902405Z RuntimeError: raised from 0 2022-11-23T01:10:27.8902588Z bar from 1 2022-11-23T01:10:27.8902842Z failed (exitcode: 1) local_rank: 0 (pid: 6690) of binary: distributed/elastic/multiprocessing/bin/echo2.py 2022-11-23T01:10:27.8903106Z ok (0.113s) 2022-11-23T01:10:27.8903579Z test_function_large_ret_val (__main__.StartProcessesTest) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:27.8903995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:27.8904421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:27.8904766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:27.8905195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:27.8905507Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:27.8905930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:27.8906273Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:27.8906686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:27.8907008Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:27.8907434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:27.8907777Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:27.8908193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:27.8908518Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:27.8908939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:27.8909269Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:27.8909522Z Closing process 6693 via signal SIGTERM 2022-11-23T01:10:27.8909743Z Closing process 6694 via signal SIGTERM 2022-11-23T01:10:27.8909958Z Closing process 6695 via signal SIGTERM 2022-11-23T01:10:27.8910135Z ok (1.964s) 2022-11-23T01:10:27.8910346Z test_function_raise (__main__.StartProcessesTest) 2022-11-23T01:10:27.8910855Z run 2x copies of echo2, raise an exception on the first ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:27.8911219Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:27.8911642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:27.8912073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:27.8912509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:27.8912823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:27.8913247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:27.8913593Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:27.8913881Z failed (exitcode: 1) local_rank: 0 (pid: 6732) of fn: echo2 (start_method: spawn) 2022-11-23T01:10:27.8914134Z Traceback (most recent call last): 2022-11-23T01:10:27.8914579Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll 2022-11-23T01:10:27.8914896Z self._pc.join(-1) 2022-11-23T01:10:27.8915234Z File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 160, in join 2022-11-23T01:10:27.8915572Z raise ProcessRaisedException(msg, error_index, failed_process.pid) 2022-11-23T01:10:27.8915910Z torch.multiprocessing.spawn.ProcessRaisedException: 2022-11-23T01:10:27.8916108Z 2022-11-23T01:10:27.8916256Z -- Process 0 terminated with the following error: 2022-11-23T01:10:27.8916491Z Traceback (most recent call last): 2022-11-23T01:10:27.8916859Z File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap 2022-11-23T01:10:27.8917144Z fn(i, *args) 2022-11-23T01:10:27.8917519Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 371, in _wrap 2022-11-23T01:10:27.8917802Z ret = record(fn)(*args_) 2022-11-23T01:10:27.8918204Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper 2022-11-23T01:10:27.8918489Z return f(*args, **kwargs) 2022-11-23T01:10:27.8918826Z File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2 2022-11-23T01:10:27.8919105Z raise RuntimeError(msg) 2022-11-23T01:10:27.8919296Z RuntimeError: hello 2022-11-23T01:10:27.8919397Z 2022-11-23T01:10:27.8919464Z ok (1.641s) 2022-11-23T01:10:27.8919701Z test_function_with_tensor (__main__.StartProcessesTest) ... ok (0.002s) 2022-11-23T01:10:27.8919996Z test_invalid_log_dir (__main__.StartProcessesTest) ... ok (0.001s) 2022-11-23T01:10:27.8920320Z test_multiprocess_context_close (__main__.StartProcessesTest) ... Closing process 6752 via signal SIGTERM 2022-11-23T01:10:27.8920736Z ok (0.009s) 2022-11-23T01:10:27.8921221Z test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn) 2022-11-23T01:10:27.8921563Z Traceback (most recent call last): 2022-11-23T01:10:27.8921949Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll 2022-11-23T01:10:27.8922265Z self._pc.join(-1) 2022-11-23T01:10:27.8922509Z File "/opt/conda/lib/python3.7/unittest/mock.py", line 1016, in __call__ 2022-11-23T01:10:27.8922756Z return _mock_self._mock_call(*args, **kwargs) 2022-11-23T01:10:27.8923022Z File "/opt/conda/lib/python3.7/unittest/mock.py", line 1076, in _mock_call 2022-11-23T01:10:27.8923249Z raise effect 2022-11-23T01:10:27.8923508Z torch.multiprocessing.spawn.ProcessRaisedException: test msg 2022-11-23T01:10:27.8923773Z ok (0.004s) 2022-11-23T01:10:27.8924234Z test_pcontext_wait (__main__.StartProcessesTest) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:27.8924612Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:27.8925026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:27.8925369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:27.8925591Z ok (2.632s) 2022-11-23T01:10:27.8925852Z test_subprocess_context_close (__main__.StartProcessesTest) ... Sending process 6763 closing signal SIGTERM 2022-11-23T01:10:27.8926111Z ok (0.010s) 2022-11-23T01:10:27.8926322Z test_to_map (__main__.StartProcessesTest) ... ok (0.002s) 2022-11-23T01:10:27.8926586Z test_validate_full_rank (__main__.StartProcessesTest) ... ok (0.001s) 2022-11-23T01:10:27.8927106Z test_void_function (__main__.StartProcessesTest) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:27.8927482Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:27.8927968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:27.8928304Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:27.8928733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:27.8929061Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:27.8929487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:27.8929817Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:27.8930078Z world 2022-11-23T01:10:27.8930226Z hello 2022-11-23T01:10:27.8930395Z Closing process 6764 via signal SIGTERM 2022-11-23T01:10:27.8930586Z ok (1.664s) 2022-11-23T01:10:27.8930799Z test_from_str_bad_input (__main__.StdTest) ... ok (0.002s) 2022-11-23T01:10:27.8931032Z test_from_value (__main__.StdTest) ... ok (0.001s) 2022-11-23T01:10:27.8931273Z test_from_value_map (__main__.StdTest) ... ok (0.001s) 2022-11-23T01:10:27.8931413Z 2022-11-23T01:10:27.8931615Z ---------------------------------------------------------------------- 2022-11-23T01:10:27.8931844Z Ran 38 tests in 19.790s 2022-11-23T01:10:27.8931961Z 2022-11-23T01:10:27.8932023Z OK 2022-11-23T01:10:27.8932112Z 2022-11-23T01:10:27.8932195Z Generating XML reports... 2022-11-23T01:10:27.8932681Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20221123011007.xml 2022-11-23T01:10:27.8933340Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20221123011007.xml 2022-11-23T01:10:27.8934000Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20221123011007.xml 2022-11-23T01:10:27.8934618Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20221123011007.xml 2022-11-23T01:10:27.8934881Z 2022-11-23T01:10:27.8935190Z ##[endgroup] 2022-11-23T01:10:27.8935701Z FINISHED PRINTING LOG FILE of distributed/elastic/multiprocessing/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-multiprocessing-api_test_6z66dnfd) 2022-11-23T01:10:27.8936004Z 2022-11-23T01:10:27.8936199Z Running distributed/_tensor/test_view_ops ... [2022-11-23 01:10:27.884272] 2022-11-23T01:10:27.8936709Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_view_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:10:27.884655] 2022-11-23T01:10:57.2872660Z 2022-11-23T01:10:57.2873230Z Expand the folded group to see the log file of distributed/_tensor/test_view_ops 2022-11-23T01:10:57.2874632Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_view_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_view_ops_0vwnw4yu) 2022-11-23T01:10:57.2875246Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_view_ops 2022-11-23T01:10:57.2875449Z 2022-11-23T01:10:57.2879137Z Running tests... 2022-11-23T01:10:57.2879730Z ---------------------------------------------------------------------- 2022-11-23T01:10:57.2880497Z test_view_groups (__main__.TestViewOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6795 2022-11-23T01:10:57.2881267Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6796 2022-11-23T01:10:57.2881774Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6797 2022-11-23T01:10:57.2882293Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6798 2022-11-23T01:10:57.2882817Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 6799 2022-11-23T01:10:57.2883314Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 6800 2022-11-23T01:10:57.2884255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:57.2884712Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:57.2885309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:57.2885791Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:57.2886241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:10:57.2886926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:57.2887544Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:57.2888216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:57.2888722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:57.2889204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:10:57.2889872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:57.2890324Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:57.2890913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:57.2891379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:57.2891824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T01:10:57.2892463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:57.2892892Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:57.2893487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:57.2893963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:57.2894392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T01:10:57.2895052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:57.2895531Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:57.2896149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:57.2896652Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:57.2897116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:10:57.2897812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:57.2898275Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:57.2898882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:57.2899371Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:57.2899842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:10:57.2900199Z ok (2.684s) 2022-11-23T01:10:57.2900614Z test_view_ops (__main__.TestViewOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6861 2022-11-23T01:10:57.2901148Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6862 2022-11-23T01:10:57.2901631Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6863 2022-11-23T01:10:57.2902182Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6864 2022-11-23T01:10:57.2902658Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 6865 2022-11-23T01:10:57.2903124Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 6866 2022-11-23T01:10:57.2903768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:57.2904246Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:57.2904903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:57.2905401Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:57.2905934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:10:57.2906588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:57.2907080Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:57.2907687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:57.2908157Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:57.2908599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T01:10:57.2909230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:57.2909677Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:57.2910298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:57.2910773Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:57.2911217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:10:57.2911957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:57.2912464Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:57.2913206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:57.2913754Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:57.2914125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:10:57.2914583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:57.2914914Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:57.2915324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:57.2915671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:57.2915989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:10:57.2916429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:10:57.2916753Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:10:57.2917176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:10:57.2917517Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:10:57.2917826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T01:10:57.2918191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:10:57.2918635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4 2022-11-23T01:10:57.2918998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:10:57.2919344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:10:57.2919705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:10:57.2920060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5 2022-11-23T01:10:57.2920542Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-11-23T01:10:57.2921330Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-11-23T01:10:57.2921851Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-11-23T01:10:57.2922364Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-11-23T01:10:57.2922863Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-11-23T01:10:57.2923368Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes. 2022-11-23T01:10:57.2923762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5 2022-11-23T01:10:57.2924170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2022-11-23T01:10:57.2924519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T01:10:57.2924873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T01:10:57.2925228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4 2022-11-23T01:10:57.2925712Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-11-23T01:10:57.2926086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-11-23T01:10:57.2926569Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-11-23T01:10:57.2927077Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-11-23T01:10:57.2927589Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-11-23T01:10:57.2927960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T01:10:57.2928319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 4 2022-11-23T01:10:57.2928674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2 2022-11-23T01:10:57.2929145Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-11-23T01:10:57.2929654Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes. 2022-11-23T01:10:57.2930042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T01:10:57.2930397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3 2022-11-23T01:10:57.2930743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 5 2022-11-23T01:10:57.2931225Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 6 nodes. 2022-11-23T01:10:57.2931810Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:3 with 6 nodes. 2022-11-23T01:10:57.2932326Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:3 with 6 nodes. 2022-11-23T01:10:57.2932822Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 6 nodes. 2022-11-23T01:10:57.2933211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 4 2022-11-23T01:10:57.2933573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2 2022-11-23T01:10:57.2934099Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 6 nodes. 2022-11-23T01:10:57.2934597Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 6 nodes. 2022-11-23T01:10:57.2934988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 5 2022-11-23T01:10:57.2935347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3 2022-11-23T01:10:57.2935705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T01:10:57.2936052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T01:10:57.2936560Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 6 nodes. 2022-11-23T01:10:57.2937395Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:4 with 6 nodes. 2022-11-23T01:10:57.2937894Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 6 nodes. 2022-11-23T01:10:57.2938405Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 6 nodes. 2022-11-23T01:10:57.2938919Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:4 with 6 nodes. 2022-11-23T01:10:57.2939427Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 6 nodes. 2022-11-23T01:10:57.2939805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 5 2022-11-23T01:10:57.2940165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0 2022-11-23T01:10:57.2940531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 4 2022-11-23T01:10:57.2940888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1 2022-11-23T01:10:57.2941226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3 2022-11-23T01:10:57.2941577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2 2022-11-23T01:10:57.2942055Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 6 nodes. 2022-11-23T01:10:57.2942564Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 6 nodes. 2022-11-23T01:10:57.2943060Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 6 nodes. 2022-11-23T01:10:57.2943451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1 2022-11-23T01:10:57.2943813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 2 2022-11-23T01:10:57.2944296Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:5 with 6 nodes. 2022-11-23T01:10:57.2944830Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 6 nodes. 2022-11-23T01:10:57.2945215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0 2022-11-23T01:10:57.2945690Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:5 with 6 nodes. 2022-11-23T01:10:57.2946064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 3 2022-11-23T01:10:57.2946425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 4 2022-11-23T01:10:57.2946823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 5 2022-11-23T01:10:57.2947304Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 6 nodes. 2022-11-23T01:10:57.2947801Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:6 with 6 nodes. 2022-11-23T01:10:57.2948311Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:6 with 6 nodes. 2022-11-23T01:10:57.2948820Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:6 with 6 nodes. 2022-11-23T01:10:57.2949322Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:6 with 6 nodes. 2022-11-23T01:10:57.2949814Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 6 nodes. 2022-11-23T01:10:57.2950102Z ok (25.092s) 2022-11-23T01:10:57.2950208Z 2022-11-23T01:10:57.2950407Z ---------------------------------------------------------------------- 2022-11-23T01:10:57.2950651Z Ran 2 tests in 27.777s 2022-11-23T01:10:57.2950754Z 2022-11-23T01:10:57.2950813Z OK 2022-11-23T01:10:57.2950908Z 2022-11-23T01:10:57.2950992Z Generating XML reports... 2022-11-23T01:10:57.2951418Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_view_ops/TEST-TestViewOps-20221123011029.xml 2022-11-23T01:10:57.2951658Z 2022-11-23T01:10:57.2952121Z ##[endgroup] 2022-11-23T01:10:57.2952568Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_view_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_view_ops_0vwnw4yu) 2022-11-23T01:10:57.2952820Z 2022-11-23T01:10:57.2953024Z Running distributed/fsdp/test_fsdp_comm_hooks ... [2022-11-23 01:10:57.287671] 2022-11-23T01:10:57.2953560Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm_hooks.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:10:57.288142] 2022-11-23T01:11:26.4586507Z 2022-11-23T01:11:26.4587074Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_comm_hooks 2022-11-23T01:11:26.4591370Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm_hooks (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_comm_hooks_2r9bsypn) 2022-11-23T01:11:26.4592648Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks 2022-11-23T01:11:26.4593154Z 2022-11-23T01:11:26.4593571Z Running tests... 2022-11-23T01:11:26.4594177Z ---------------------------------------------------------------------- 2022-11-23T01:11:26.4594944Z test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:11:26.4596129Z test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:11:26.4597266Z test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:11:26.4598166Z test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:11:26.4598956Z test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:11:26.4599813Z test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:11:26.4600956Z test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) 2022-11-23T01:11:26.4601934Z Tests FSDP's default communication hook's behavior and correctness. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7082 2022-11-23T01:11:26.4602597Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7083 2022-11-23T01:11:26.4603102Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7084 2022-11-23T01:11:26.4603621Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7085 2022-11-23T01:11:26.4604389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4604924Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4605578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4606056Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4606493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:11:26.4607161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4607622Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4608260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4608754Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4609283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:11:26.4610006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4610486Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4611152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4611628Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4612099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:11:26.4612743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4613190Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4613802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4614256Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4614713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:11:26.4615412Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4616153Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4617070Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4617837Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4618405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:11:26.4618916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:11:26.4619422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:11:26.4619998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:11:26.4620374Z dist init r=3, world=4 2022-11-23T01:11:26.4620624Z dist init r=0, world=4 2022-11-23T01:11:26.4620881Z dist init r=1, world=4 2022-11-23T01:11:26.4621142Z dist init r=2, world=4 2022-11-23T01:11:26.4621429Z skip: Need at least 2 CUDA devices (2.196s) 2022-11-23T01:11:26.4621937Z test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) 2022-11-23T01:11:26.4622765Z Tests FSDP's default communication hook's behavior and correctness. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7138 2022-11-23T01:11:26.4623360Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7139 2022-11-23T01:11:26.4623819Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7140 2022-11-23T01:11:26.4624299Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7141 2022-11-23T01:11:26.4624974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4625462Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4626077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4626607Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4627090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:11:26.4627773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4628231Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4628873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4629367Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4629806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:11:26.4630464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4630907Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4631515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4632086Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4632569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:11:26.4633276Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4633992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4634529Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4635252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4635791Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4636122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:11:26.4636629Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4637155Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4637672Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4638080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:11:26.4638432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:11:26.4638782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:11:26.4639115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:11:26.4639373Z dist init r=2, world=4 2022-11-23T01:11:26.4639554Z dist init r=0, world=4 2022-11-23T01:11:26.4639724Z dist init r=1, world=4 2022-11-23T01:11:26.4639908Z dist init r=3, world=4 2022-11-23T01:11:26.4640118Z skip: Need at least 2 CUDA devices (1.822s) 2022-11-23T01:11:26.4640469Z test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) 2022-11-23T01:11:26.4641203Z Tests FSDP's default communication hook's behavior and correctness. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7194 2022-11-23T01:11:26.4641614Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7195 2022-11-23T01:11:26.4641940Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7196 2022-11-23T01:11:26.4642264Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7197 2022-11-23T01:11:26.4642710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4643048Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4643472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4643822Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4644157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:11:26.4644616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4644947Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4645376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4645721Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4646041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:11:26.4646507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4646835Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4647260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4647590Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4647921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:11:26.4648479Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4648964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4649291Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4649714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4650057Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4650381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:11:26.4650871Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4651433Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4651953Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4652325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:11:26.4652673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:11:26.4653015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:11:26.4653349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:11:26.4653610Z dist init r=0, world=4 2022-11-23T01:11:26.4653792Z dist init r=2, world=4 2022-11-23T01:11:26.4653971Z dist init r=3, world=4 2022-11-23T01:11:26.4654134Z dist init r=1, world=4 2022-11-23T01:11:26.4654344Z skip: Need at least 2 CUDA devices (1.722s) 2022-11-23T01:11:26.4654712Z test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) 2022-11-23T01:11:26.4655264Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7250 2022-11-23T01:11:26.4655646Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7251 2022-11-23T01:11:26.4655968Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7252 2022-11-23T01:11:26.4656293Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7253 2022-11-23T01:11:26.4656730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4657061Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4657490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4657823Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4658162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:11:26.4658692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4659261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4659722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4660070Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4660402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:11:26.4660864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4661192Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4661663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4662011Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4662329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:11:26.4662804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4663130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4663552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4663918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4664257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:11:26.4664748Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4665259Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4665776Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4666296Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4666687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:11:26.4667024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:11:26.4667366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:11:26.4667705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:11:26.4667966Z dist init r=0, world=4 2022-11-23T01:11:26.4668137Z dist init r=2, world=4 2022-11-23T01:11:26.4668315Z dist init r=1, world=4 2022-11-23T01:11:26.4668492Z dist init r=3, world=4 2022-11-23T01:11:26.4668683Z skip: Need at least 2 CUDA devices (1.821s) 2022-11-23T01:11:26.4669049Z test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) 2022-11-23T01:11:26.4669609Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7306 2022-11-23T01:11:26.4669977Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7307 2022-11-23T01:11:26.4670306Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7308 2022-11-23T01:11:26.4670624Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7309 2022-11-23T01:11:26.4671074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4671392Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4671819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4672235Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4672564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:11:26.4673038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4673371Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4673801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4674132Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4674501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:11:26.4674967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4675291Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4675701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4676042Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4676373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:11:26.4676884Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4677384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4677710Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4678135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4678470Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4678806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:11:26.4679303Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4679820Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4680326Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4680820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:11:26.4681178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:11:26.4681525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:11:26.4681862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:11:26.4682121Z dist init r=2, world=4 2022-11-23T01:11:26.4682303Z dist init r=3, world=4 2022-11-23T01:11:26.4682471Z dist init r=0, world=4 2022-11-23T01:11:26.4682648Z dist init r=1, world=4 2022-11-23T01:11:26.4682854Z skip: Need at least 2 CUDA devices (1.822s) 2022-11-23T01:11:26.4683223Z test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) 2022-11-23T01:11:26.4683795Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7362 2022-11-23T01:11:26.4684183Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7363 2022-11-23T01:11:26.4684508Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7364 2022-11-23T01:11:26.4684819Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7365 2022-11-23T01:11:26.4685262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4685590Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4686005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4686354Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4686689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:11:26.4687159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4687547Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4687978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4688324Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4688646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:11:26.4689110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4689434Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4689898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4690232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4690565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:11:26.4691030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4691360Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4691770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4692113Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4692440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:11:26.4692918Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4693435Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4693954Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4694466Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4694837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:11:26.4695181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:11:26.4695515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:11:26.4695860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:11:26.4696107Z dist init r=2, world=4 2022-11-23T01:11:26.4696291Z dist init r=1, world=4 2022-11-23T01:11:26.4696468Z dist init r=0, world=4 2022-11-23T01:11:26.4696633Z dist init r=3, world=4 2022-11-23T01:11:26.4696840Z skip: Need at least 2 CUDA devices (1.921s) 2022-11-23T01:11:26.4697210Z test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) 2022-11-23T01:11:26.4697760Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7418 2022-11-23T01:11:26.4698145Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7419 2022-11-23T01:11:26.4698473Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7420 2022-11-23T01:11:26.4698793Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7421 2022-11-23T01:11:26.4699237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4699565Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4700027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4700383Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4700709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:11:26.4701177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4701503Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4701917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4702263Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4702631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:11:26.4703093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4703407Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4703833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4704175Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4704501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:11:26.4704959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4705286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4705722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4706056Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4706398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:11:26.4706895Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4707413Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4707920Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4708430Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4708819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:11:26.4709162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:11:26.4709497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:11:26.4709837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:11:26.4710096Z dist init r=3, world=4 2022-11-23T01:11:26.4710265Z dist init r=0, world=4 2022-11-23T01:11:26.4710443Z dist init r=1, world=4 2022-11-23T01:11:26.4710621Z dist init r=2, world=4 2022-11-23T01:11:26.4710813Z skip: Need at least 2 CUDA devices (1.822s) 2022-11-23T01:11:26.4711176Z test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) 2022-11-23T01:11:26.4711727Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7474 2022-11-23T01:11:26.4712189Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7475 2022-11-23T01:11:26.4712502Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7476 2022-11-23T01:11:26.4712850Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7477 2022-11-23T01:11:26.4713305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4713620Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4714049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4714394Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4714733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:11:26.4715222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4715552Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4715983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4716330Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4716653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:11:26.4717124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4717449Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4717863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4718205Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4718542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:11:26.4719004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4719323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4719749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4720089Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4720412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:11:26.4721069Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4721588Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4722107Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4722608Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4722990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:11:26.4723340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:11:26.4723682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:11:26.4724014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:11:26.4724269Z dist init r=0, world=4 2022-11-23T01:11:26.4724448Z dist init r=3, world=4 2022-11-23T01:11:26.4724617Z dist init r=1, world=4 2022-11-23T01:11:26.4724795Z dist init r=2, world=4 2022-11-23T01:11:26.4724998Z skip: Need at least 2 CUDA devices (1.921s) 2022-11-23T01:11:26.4725355Z test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) 2022-11-23T01:11:26.4725976Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7530 2022-11-23T01:11:26.4726367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7531 2022-11-23T01:11:26.4726690Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7532 2022-11-23T01:11:26.4727002Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7533 2022-11-23T01:11:26.4727448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4727780Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4728254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4728590Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4728925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:11:26.4729395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4729713Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4730141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4730487Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4730817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:11:26.4731269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4731593Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4732039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4732408Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4732741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:11:26.4733210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4733533Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4733942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4734284Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4734620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:11:26.4735119Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4735626Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4736146Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4736666Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4737051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:11:26.4737387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:11:26.4737729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:11:26.4738072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:11:26.4738319Z dist init r=3, world=4 2022-11-23T01:11:26.4738541Z dist init r=1, world=4 2022-11-23T01:11:26.4738726Z dist init r=2, world=4 2022-11-23T01:11:26.4738893Z dist init r=0, world=4 2022-11-23T01:11:26.4739098Z skip: Need at least 2 CUDA devices (1.821s) 2022-11-23T01:11:26.4739483Z test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:11:26.4739999Z test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:11:26.4740641Z test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-11-23T01:11:26.4741185Z test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-11-23T01:11:26.4741694Z test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-11-23T01:11:26.4742195Z test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-11-23T01:11:26.4742654Z test_registering_hook_non_root_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) 2022-11-23T01:11:26.4743192Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7586 2022-11-23T01:11:26.4743589Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7587 2022-11-23T01:11:26.4743904Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7588 2022-11-23T01:11:26.4744227Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7589 2022-11-23T01:11:26.4744680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4745000Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4745429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4745786Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4746123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:11:26.4746583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4746913Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4747345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4747684Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4748023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:11:26.4748491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4748818Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4749232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4749579Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4749914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:11:26.4750404Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4750919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4751249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4751675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4752076Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4752409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:11:26.4752905Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4753458Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4753962Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4754346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:11:26.4754697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:11:26.4755044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:11:26.4755375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:11:26.4755629Z dist init r=1, world=4 2022-11-23T01:11:26.4755810Z dist init r=2, world=4 2022-11-23T01:11:26.4766913Z dist init r=0, world=4 2022-11-23T01:11:26.4767227Z dist init r=3, world=4 2022-11-23T01:11:26.4767457Z skip: Need at least 2 CUDA devices (1.721s) 2022-11-23T01:11:26.4767797Z test_registering_hook_non_root_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) 2022-11-23T01:11:26.4768376Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7642 2022-11-23T01:11:26.4768774Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7643 2022-11-23T01:11:26.4769110Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7644 2022-11-23T01:11:26.4769427Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7645 2022-11-23T01:11:26.4769889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4770234Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4770667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4771008Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4771348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:11:26.4771824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4772156Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4772573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4772923Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4773264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:11:26.4773718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4774047Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4774477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4774946Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4775271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:11:26.4775779Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4776277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4776609Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4777025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4777417Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4777756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:11:26.4778241Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4778762Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4779278Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4779673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:11:26.4780011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:11:26.4780358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:11:26.4780706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:11:26.4780969Z dist init r=1, world=4 2022-11-23T01:11:26.4781143Z dist init r=2, world=4 2022-11-23T01:11:26.4781325Z dist init r=3, world=4 2022-11-23T01:11:26.4781505Z dist init r=0, world=4 2022-11-23T01:11:26.4781700Z skip: Need at least 2 CUDA devices (1.720s) 2022-11-23T01:11:26.4782035Z test_registering_hook_non_root_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) 2022-11-23T01:11:26.4782582Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7698 2022-11-23T01:11:26.4782961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7699 2022-11-23T01:11:26.4783289Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7700 2022-11-23T01:11:26.4783622Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7701 2022-11-23T01:11:26.4784075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4784393Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4784828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4785175Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4785497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:11:26.4785969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4786299Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4786722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4787055Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4787392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:11:26.4787890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4788218Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4788629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4788974Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4789312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:11:26.4789764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4790124Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4790555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4790903Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4791223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:11:26.4791721Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4792328Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4792844Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4793350Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4793745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:11:26.4794094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:11:26.4794432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:11:26.4794779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:11:26.4795040Z dist init r=2, world=4 2022-11-23T01:11:26.4795225Z dist init r=3, world=4 2022-11-23T01:11:26.4795390Z dist init r=1, world=4 2022-11-23T01:11:26.4795572Z dist init r=0, world=4 2022-11-23T01:11:26.4795780Z skip: Need at least 2 CUDA devices (1.822s) 2022-11-23T01:11:26.4796103Z test_registering_hook_submodules_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) 2022-11-23T01:11:26.4796647Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7754 2022-11-23T01:11:26.4797040Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7755 2022-11-23T01:11:26.4797361Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7756 2022-11-23T01:11:26.4797692Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7757 2022-11-23T01:11:26.4798143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4798477Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4798894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4799242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4799583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:11:26.4800057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4800374Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4801037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4801389Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4801718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:11:26.4802192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4802521Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4802946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4803323Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4803660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:11:26.4804129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4804446Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4804872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4805221Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4805558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:11:26.4806042Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4806568Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4807091Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4807606Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4807975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:11:26.4808328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:11:26.4808677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:11:26.4809027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:11:26.4809276Z dist init r=1, world=4 2022-11-23T01:11:26.4809460Z dist init r=2, world=4 2022-11-23T01:11:26.4809640Z dist init r=0, world=4 2022-11-23T01:11:26.4809807Z dist init r=3, world=4 2022-11-23T01:11:26.4810020Z skip: Need at least 2 CUDA devices (1.821s) 2022-11-23T01:11:26.4810355Z test_registering_hook_submodules_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) 2022-11-23T01:11:26.4810882Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7810 2022-11-23T01:11:26.4811275Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7811 2022-11-23T01:11:26.4811608Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7812 2022-11-23T01:11:26.4811936Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7813 2022-11-23T01:11:26.4812371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4812708Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4813141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4813475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4813862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:11:26.4814336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4814666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4815080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4815432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4815768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:11:26.4816252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4816577Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4817010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4817358Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4817680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:11:26.4818146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4818482Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4818913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4819247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4819586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:11:26.4820083Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4820593Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4821109Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4821630Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4822016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:11:26.4822358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:11:26.4822704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:11:26.4823049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:11:26.4823312Z dist init r=1, world=4 2022-11-23T01:11:26.4823482Z dist init r=0, world=4 2022-11-23T01:11:26.4823665Z dist init r=3, world=4 2022-11-23T01:11:26.4823845Z dist init r=2, world=4 2022-11-23T01:11:26.4824039Z skip: Need at least 2 CUDA devices (1.721s) 2022-11-23T01:11:26.4824378Z test_registering_hook_submodules_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) 2022-11-23T01:11:26.4824924Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7866 2022-11-23T01:11:26.4825415Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7867 2022-11-23T01:11:26.4825754Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7868 2022-11-23T01:11:26.4826078Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7869 2022-11-23T01:11:26.4826573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4826892Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4827321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4827670Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4827995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:11:26.4828466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4828799Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4829261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4829592Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4829933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:11:26.4830432Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4830929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4831246Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4831674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4832072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4832398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:11:26.4832870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:26.4833208Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:26.4833635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:26.4833966Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:26.4834304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:11:26.4834803Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4835325Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4835880Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:26.4836380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:11:26.4836735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:11:26.4837068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:11:26.4837417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:11:26.4837678Z dist init r=3, world=4 2022-11-23T01:11:26.4837861Z dist init r=1, world=4 2022-11-23T01:11:26.4838029Z dist init r=2, world=4 2022-11-23T01:11:26.4838211Z dist init r=0, world=4 2022-11-23T01:11:26.4838420Z skip: Need at least 2 CUDA devices (1.822s) 2022-11-23T01:11:26.4838549Z 2022-11-23T01:11:26.4838755Z ---------------------------------------------------------------------- 2022-11-23T01:11:26.4838998Z Ran 27 tests in 27.505s 2022-11-23T01:11:26.4839116Z 2022-11-23T01:11:26.4839192Z OK (skipped=27) 2022-11-23T01:11:26.4839301Z 2022-11-23T01:11:26.4839373Z Generating XML reports... 2022-11-23T01:11:26.4839890Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks/TEST-TestCommunicationHooks-20221123011058.xml 2022-11-23T01:11:26.4840166Z 2022-11-23T01:11:26.4840716Z ##[endgroup] 2022-11-23T01:11:26.4841207Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm_hooks (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_comm_hooks_2r9bsypn) 2022-11-23T01:11:26.4841476Z 2022-11-23T01:11:26.4841667Z Running distributed/fsdp/test_fsdp_checkpoint ... [2022-11-23 01:11:26.459858] 2022-11-23T01:11:26.4842200Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:11:26.460304] 2022-11-23T01:11:56.5800170Z 2022-11-23T01:11:56.5826720Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_checkpoint 2022-11-23T01:11:56.5828087Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_checkpoint_gaqtxtg3) 2022-11-23T01:11:56.5829164Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint 2022-11-23T01:11:56.5829550Z 2022-11-23T01:11:56.5829697Z Running tests... 2022-11-23T01:11:56.5830191Z ---------------------------------------------------------------------- 2022-11-23T01:11:56.5830952Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7933 2022-11-23T01:11:56.5831727Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7934 2022-11-23T01:11:56.5832337Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7935 2022-11-23T01:11:56.5832823Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7936 2022-11-23T01:11:56.5833581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5834124Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5834812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5835356Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5835889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:11:56.5836634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5837134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5837890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5838461Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5839002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:11:56.5839786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5840391Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5841435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5842029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5842630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:11:56.5843516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5844085Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5845085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5845701Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5846229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:11:56.5847051Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5847925Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5848763Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5849762Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5850329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:11:56.5850861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:11:56.5851397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:11:56.5851908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:11:56.5852306Z dist init r=2, world=4 2022-11-23T01:11:56.5852584Z dist init r=1, world=4 2022-11-23T01:11:56.5852835Z dist init r=0, world=4 2022-11-23T01:11:56.5853084Z dist init r=3, world=4 2022-11-23T01:11:56.5853384Z skip: Need at least 2 CUDA devices (2.083s) 2022-11-23T01:11:56.5854088Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7989 2022-11-23T01:11:56.5854848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7990 2022-11-23T01:11:56.5855347Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7991 2022-11-23T01:11:56.5855863Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7992 2022-11-23T01:11:56.5856617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5857084Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5857751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5858260Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5858683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:11:56.5859149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5859483Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5860163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5860495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5860826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:11:56.5861291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5861618Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5862030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5862373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5862794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:11:56.5863264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5863576Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5864000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5864341Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5864662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:11:56.5865196Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5865717Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5866233Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5866736Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5867119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:11:56.5867463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:11:56.5867804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:11:56.5868133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:11:56.5868391Z dist init r=1, world=4 2022-11-23T01:11:56.5868573Z dist init r=3, world=4 2022-11-23T01:11:56.5868735Z dist init r=0, world=4 2022-11-23T01:11:56.5868909Z dist init r=2, world=4 2022-11-23T01:11:56.5869112Z skip: Need at least 2 CUDA devices (1.822s) 2022-11-23T01:11:56.5869555Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8045 2022-11-23T01:11:56.5870038Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8046 2022-11-23T01:11:56.5870360Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8047 2022-11-23T01:11:56.5870678Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8048 2022-11-23T01:11:56.5871113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5871443Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5871952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5872304Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5872626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:11:56.5873096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5873422Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5873833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5874173Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5874511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:11:56.5874975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5875283Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5875741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5876086Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5876407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:11:56.5876867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5877192Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5877615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5877974Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5878303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:11:56.5878796Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5879312Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5879814Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5880329Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5880868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:11:56.5881223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:11:56.5881553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:11:56.5881892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:11:56.5882152Z dist init r=2, world=4 2022-11-23T01:11:56.5882319Z dist init r=3, world=4 2022-11-23T01:11:56.5882497Z dist init r=1, world=4 2022-11-23T01:11:56.5882672Z dist init r=0, world=4 2022-11-23T01:11:56.5882864Z skip: Need at least 2 CUDA devices (1.722s) 2022-11-23T01:11:56.5883323Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8101 2022-11-23T01:11:56.5883806Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8102 2022-11-23T01:11:56.5884129Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8103 2022-11-23T01:11:56.5884435Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8104 2022-11-23T01:11:56.5884887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5885216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5885630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5885972Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5886307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:11:56.5886770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5887082Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5887504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5887844Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5888245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:11:56.5888702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5889026Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5889452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5889781Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5890115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:11:56.5890625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5890947Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5891360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5891698Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5892028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:11:56.5892517Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5893022Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5893532Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5894049Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5894418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:11:56.5894759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:11:56.5895093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:11:56.5895433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:11:56.5895676Z dist init r=0, world=4 2022-11-23T01:11:56.5895855Z dist init r=2, world=4 2022-11-23T01:11:56.5896030Z dist init r=3, world=4 2022-11-23T01:11:56.5896192Z dist init r=1, world=4 2022-11-23T01:11:56.5896395Z skip: Need at least 2 CUDA devices (1.822s) 2022-11-23T01:11:56.5896852Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8157 2022-11-23T01:11:56.5897333Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8158 2022-11-23T01:11:56.5897646Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8159 2022-11-23T01:11:56.5897960Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8160 2022-11-23T01:11:56.5898407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5898721Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5899144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5899486Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5899824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:11:56.5900275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5900629Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5901056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5901394Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5901713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:11:56.5902179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5902500Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5902945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5903289Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5903624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:11:56.5904089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5904399Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5904821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5905160Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5905478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:11:56.5905966Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5906483Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5906997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5907499Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5907883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:11:56.5908229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:11:56.5908569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:11:56.5908895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:11:56.5909151Z dist init r=3, world=4 2022-11-23T01:11:56.5909329Z dist init r=2, world=4 2022-11-23T01:11:56.5909491Z dist init r=0, world=4 2022-11-23T01:11:56.5909663Z dist init r=1, world=4 2022-11-23T01:11:56.5909865Z skip: Need at least 2 CUDA devices (1.722s) 2022-11-23T01:11:56.5910312Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8213 2022-11-23T01:11:56.5910790Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8214 2022-11-23T01:11:56.5911110Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8215 2022-11-23T01:11:56.5911430Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8216 2022-11-23T01:11:56.5911930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5912265Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5912694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5913077Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5913401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:11:56.5913866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5914193Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5914604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5914946Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5915277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:11:56.5915815Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5916294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5916618Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5917038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5917380Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5917700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:11:56.5918164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5918491Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5918898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5919236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5919570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:11:56.5920064Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5920569Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5921231Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5921617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:11:56.5921968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:11:56.5922296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:11:56.5922631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:11:56.5922888Z dist init r=3, world=4 2022-11-23T01:11:56.5923055Z dist init r=0, world=4 2022-11-23T01:11:56.5923230Z dist init r=2, world=4 2022-11-23T01:11:56.5923402Z dist init r=1, world=4 2022-11-23T01:11:56.5923591Z skip: Need at least 2 CUDA devices (1.721s) 2022-11-23T01:11:56.5924044Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8269 2022-11-23T01:11:56.5924523Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8270 2022-11-23T01:11:56.5924847Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8271 2022-11-23T01:11:56.5925152Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8272 2022-11-23T01:11:56.5925661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5925996Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5926422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5926755Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5927092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:11:56.5927558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5927873Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5928340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5928679Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5929010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:11:56.5929457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5929780Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5930200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5930525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5930852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:11:56.5931348Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5931847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5932161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5932586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5932926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5933256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:11:56.5933734Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5934245Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5934759Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5935138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:11:56.5935473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:11:56.5935809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:11:56.5936151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:11:56.5936394Z dist init r=0, world=4 2022-11-23T01:11:56.5936573Z dist init r=3, world=4 2022-11-23T01:11:56.5936746Z dist init r=1, world=4 2022-11-23T01:11:56.5936911Z dist init r=2, world=4 2022-11-23T01:11:56.5937113Z skip: Need at least 2 CUDA devices (1.822s) 2022-11-23T01:11:56.5937569Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8325 2022-11-23T01:11:56.5938050Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8326 2022-11-23T01:11:56.5938396Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8327 2022-11-23T01:11:56.5938713Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8328 2022-11-23T01:11:56.5939161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5939477Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5939901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5940245Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5940613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:11:56.5941063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5941393Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5941818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5942158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5942477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:11:56.5942938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5943259Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5943671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5944006Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5944342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:11:56.5944809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5945119Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5945542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5945886Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5946206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:11:56.5946693Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5947210Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5947726Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5948222Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5948603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:11:56.5948948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:11:56.5949289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:11:56.5949618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:11:56.5949871Z dist init r=1, world=4 2022-11-23T01:11:56.5950050Z dist init r=0, world=4 2022-11-23T01:11:56.5950214Z dist init r=2, world=4 2022-11-23T01:11:56.5950386Z dist init r=3, world=4 2022-11-23T01:11:56.5950589Z skip: Need at least 2 CUDA devices (1.722s) 2022-11-23T01:11:56.5951076Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8381 2022-11-23T01:11:56.5951548Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8382 2022-11-23T01:11:56.5951957Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8383 2022-11-23T01:11:56.5952290Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8384 2022-11-23T01:11:56.5952730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5953097Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5953530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5953875Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5954197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:11:56.5954660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5954988Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5955413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5955740Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5956076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:11:56.5956542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5956854Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5957280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5957623Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5957954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:11:56.5958433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5958923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5959248Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5959659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5959999Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5960333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:11:56.5960980Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5961485Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5962001Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5962382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:11:56.5962727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:11:56.5963058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:11:56.5963395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:11:56.5963710Z dist init r=1, world=4 2022-11-23T01:11:56.5963880Z dist init r=3, world=4 2022-11-23T01:11:56.5964058Z dist init r=0, world=4 2022-11-23T01:11:56.5964233Z dist init r=2, world=4 2022-11-23T01:11:56.5964425Z skip: Need at least 2 CUDA devices (1.721s) 2022-11-23T01:11:56.5964883Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8437 2022-11-23T01:11:56.5965367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8438 2022-11-23T01:11:56.5965733Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8439 2022-11-23T01:11:56.5966039Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8440 2022-11-23T01:11:56.5966485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5966817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5967246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5967577Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5967913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:11:56.5968380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5968695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5969121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5969462Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5969793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:11:56.5970274Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5970765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5971089Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5971512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5971841Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5972176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:11:56.5972640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5972953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5973373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5973712Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5974043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:11:56.5974523Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5975035Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5975550Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5975930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:11:56.5976291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:11:56.5976629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:11:56.5976971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:11:56.5977215Z dist init r=1, world=4 2022-11-23T01:11:56.5977391Z dist init r=3, world=4 2022-11-23T01:11:56.5977567Z dist init r=2, world=4 2022-11-23T01:11:56.5977729Z dist init r=0, world=4 2022-11-23T01:11:56.5977932Z skip: Need at least 2 CUDA devices (1.821s) 2022-11-23T01:11:56.5978386Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8493 2022-11-23T01:11:56.5978907Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8494 2022-11-23T01:11:56.5979220Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8495 2022-11-23T01:11:56.5979535Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8496 2022-11-23T01:11:56.5979984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5980311Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5980727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5981073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5981411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:11:56.5981862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5982189Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5982614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5982956Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5983276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:11:56.5983742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5984064Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5984476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5984815Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5985146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:11:56.5985640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5986124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5986449Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5986870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5987211Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5987526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:11:56.5988018Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5988565Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5989078Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5989443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:11:56.5989787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:11:56.5990131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:11:56.5990460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:11:56.5990741Z dist init r=0, world=4 2022-11-23T01:11:56.5990918Z dist init r=2, world=4 2022-11-23T01:11:56.5991082Z dist init r=1, world=4 2022-11-23T01:11:56.5991257Z dist init r=3, world=4 2022-11-23T01:11:56.5991460Z skip: Need at least 2 CUDA devices (1.721s) 2022-11-23T01:11:56.5991993Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8549 2022-11-23T01:11:56.5992474Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8550 2022-11-23T01:11:56.5992802Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8551 2022-11-23T01:11:56.5993123Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8552 2022-11-23T01:11:56.5993563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5993892Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5994317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5994661Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5994984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:11:56.5995449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5995775Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5996197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5996525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5996858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:11:56.5997318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.5997627Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.5998051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.5998394Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.5998727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:11:56.5999206Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.5999701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.6000024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.6000448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.6000904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.6001306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:11:56.6001808Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.6002314Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.6002829Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.6003216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:11:56.6003566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:11:56.6003953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:11:56.6004293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:11:56.6004553Z dist init r=1, world=4 2022-11-23T01:11:56.6004721Z dist init r=0, world=4 2022-11-23T01:11:56.6004898Z dist init r=2, world=4 2022-11-23T01:11:56.6005073Z dist init r=3, world=4 2022-11-23T01:11:56.6005266Z skip: Need at least 2 CUDA devices (1.721s) 2022-11-23T01:11:56.6005724Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8605 2022-11-23T01:11:56.6006215Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8606 2022-11-23T01:11:56.6006538Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8607 2022-11-23T01:11:56.6006849Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8608 2022-11-23T01:11:56.6007298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.6007632Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.6008058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.6008393Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.6008728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:11:56.6009198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.6009521Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.6009936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.6010279Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.6010610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:11:56.6011060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.6011382Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.6011803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.6012145Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.6012465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:11:56.6012928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.6013252Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.6013661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.6014042Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.6014376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:11:56.6014870Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.6015373Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.6015888Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.6016438Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.6016818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:11:56.6017156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:11:56.6017491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:11:56.6017836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:11:56.6018082Z dist init r=1, world=4 2022-11-23T01:11:56.6018259Z dist init r=2, world=4 2022-11-23T01:11:56.6018434Z dist init r=3, world=4 2022-11-23T01:11:56.6018599Z dist init r=0, world=4 2022-11-23T01:11:56.6018800Z skip: Need at least 2 CUDA devices (1.722s) 2022-11-23T01:11:56.6019252Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8661 2022-11-23T01:11:56.6019733Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8662 2022-11-23T01:11:56.6020048Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8663 2022-11-23T01:11:56.6020362Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8664 2022-11-23T01:11:56.6020809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.6021136Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.6021548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.6021891Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.6022228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:11:56.6022684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.6023008Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.6023435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.6023776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.6024097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:11:56.6024558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.6024883Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.6025307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.6025636Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.6025968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:11:56.6026461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.6026774Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.6027196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.6027535Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.6027865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:11:56.6028338Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.6028888Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.6029403Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.6029908Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.6030275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:11:56.6030619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:11:56.6030960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:11:56.6031287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:11:56.6031542Z dist init r=1, world=4 2022-11-23T01:11:56.6031723Z dist init r=0, world=4 2022-11-23T01:11:56.6031962Z dist init r=2, world=4 2022-11-23T01:11:56.6032139Z dist init r=3, world=4 2022-11-23T01:11:56.6032342Z skip: Need at least 2 CUDA devices (1.722s) 2022-11-23T01:11:56.6032798Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8717 2022-11-23T01:11:56.6033273Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8718 2022-11-23T01:11:56.6033598Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8719 2022-11-23T01:11:56.6033918Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8720 2022-11-23T01:11:56.6034373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.6034690Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.6035114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.6035457Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.6035784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:11:56.6036250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.6036577Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.6037003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.6037334Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.6037666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:11:56.6038128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.6038439Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.6038939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.6039284Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.6039620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:11:56.6040099Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.6040704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.6041128Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.6041621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.6041950Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.6042286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:11:56.6042782Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.6043286Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.6043803Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.6044192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:11:56.6044536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:11:56.6044868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:11:56.6045204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:11:56.6045460Z dist init r=0, world=4 2022-11-23T01:11:56.6045640Z dist init r=2, world=4 2022-11-23T01:11:56.6045804Z dist init r=1, world=4 2022-11-23T01:11:56.6045977Z dist init r=3, world=4 2022-11-23T01:11:56.6046181Z skip: Need at least 2 CUDA devices (1.922s) 2022-11-23T01:11:56.6046623Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8773 2022-11-23T01:11:56.6047103Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8774 2022-11-23T01:11:56.6047430Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8775 2022-11-23T01:11:56.6047739Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8776 2022-11-23T01:11:56.6048188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.6048519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.6048951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.6049282Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.6049617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:11:56.6050082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.6050408Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.6050821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.6051168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.6051499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:11:56.6052018Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.6052512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.6052838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.6053262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.6053592Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.6053927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:11:56.6054424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:11:56.6054746Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:11:56.6055160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:11:56.6055502Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:11:56.6055832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:11:56.6056308Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.6056825Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.6057338Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:11:56.6057719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:11:56.6058056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:11:56.6058397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:11:56.6058737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:11:56.6058994Z dist init r=1, world=4 2022-11-23T01:11:56.6059160Z dist init r=3, world=4 2022-11-23T01:11:56.6059335Z dist init r=2, world=4 2022-11-23T01:11:56.6059508Z dist init r=0, world=4 2022-11-23T01:11:56.6059698Z skip: Need at least 2 CUDA devices (1.721s) 2022-11-23T01:11:56.6059835Z 2022-11-23T01:11:56.6060034Z ---------------------------------------------------------------------- 2022-11-23T01:11:56.6060276Z Ran 16 tests in 28.510s 2022-11-23T01:11:56.6060388Z 2022-11-23T01:11:56.6060447Z OK (skipped=16) 2022-11-23T01:11:56.6060552Z 2022-11-23T01:11:56.6060636Z Generating XML reports... 2022-11-23T01:11:56.6061099Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint/TEST-TestFSDPCheckpoint-20221123011127.xml 2022-11-23T01:11:56.6061371Z 2022-11-23T01:11:56.6061678Z ##[endgroup] 2022-11-23T01:11:56.6062147Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_checkpoint_gaqtxtg3) 2022-11-23T01:11:56.6062418Z 2022-11-23T01:11:56.6062634Z Running distributed/fsdp/test_fsdp_use_orig_params ... [2022-11-23 01:11:56.581008] 2022-11-23T01:11:56.6063168Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_use_orig_params.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:11:56.581350] 2022-11-23T01:12:28.0342837Z 2022-11-23T01:12:28.0343580Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_use_orig_params 2022-11-23T01:12:28.0344703Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_use_orig_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_use_orig_params_l6yo7x99) 2022-11-23T01:12:28.0345898Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params 2022-11-23T01:12:28.0401526Z 2022-11-23T01:12:28.0401990Z Running tests... 2022-11-23T01:12:28.0402625Z ---------------------------------------------------------------------- 2022-11-23T01:12:28.0403314Z test_named_parameters_in_forward (__main__.TestFSDPUseOrigParamsFQNs) 2022-11-23T01:12:28.0403954Z Tests that calling ``named_parameters()`` during forward returns FQNs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8840 2022-11-23T01:12:28.0404590Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8841 2022-11-23T01:12:28.0405394Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8842 2022-11-23T01:12:28.0405721Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8843 2022-11-23T01:12:28.0406186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0406525Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0406963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0407303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0407642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:12:28.0408113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0408439Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0408853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0409196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0409534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:12:28.0409990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0410315Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0410738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0411085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0411402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:12:28.0411874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0412202Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0412631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0412957Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0413288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:12:28.0413780Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:12:28.0414288Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:12:28.0414808Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:12:28.0415323Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:12:28.0415710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:12:28.0416121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:12:28.0416463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:12:28.0416809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:12:28.0417068Z dist init r=1, world=4 2022-11-23T01:12:28.0417238Z dist init r=3, world=4 2022-11-23T01:12:28.0417418Z dist init r=2, world=4 2022-11-23T01:12:28.0417595Z dist init r=0, world=4 2022-11-23T01:12:28.0417787Z skip: Need at least 2 CUDA devices (2.084s) 2022-11-23T01:12:28.0418064Z test_param_and_buffer_names (__main__.TestFSDPUseOrigParamsFQNs) 2022-11-23T01:12:28.0418493Z Tests that, for ``use_orig_params=True``, the parameter and buffer ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8896 2022-11-23T01:12:28.0418870Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8897 2022-11-23T01:12:28.0419196Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8898 2022-11-23T01:12:28.0419517Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8899 2022-11-23T01:12:28.0419968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0420285Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0420716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0421064Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0421392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:12:28.0421860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0422187Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0422611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0422940Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0423273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:12:28.0423737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0424063Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0424474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0424822Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0425144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:12:28.0425613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0425934Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0426341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0426680Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0427007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:12:28.0427493Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:12:28.0428005Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:12:28.0428551Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:12:28.0429069Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:12:28.0429454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:12:28.0429787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:12:28.0430123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:12:28.0430463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:12:28.0430780Z dist init r=3, world=4 2022-11-23T01:12:28.0430948Z dist init r=0, world=4 2022-11-23T01:12:28.0431125Z dist init r=1, world=4 2022-11-23T01:12:28.0431299Z dist init r=2, world=4 2022-11-23T01:12:28.0431491Z skip: Need at least 2 CUDA devices (1.820s) 2022-11-23T01:12:28.0431854Z test_diff_hyperparams_cpu_offload_sharding_strategy_str_full_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups) 2022-11-23T01:12:28.0432419Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8952 2022-11-23T01:12:28.0432805Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8953 2022-11-23T01:12:28.0433259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0433588Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0434015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0434350Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0434684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:12:28.0435150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0435463Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0435885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0436225Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0436554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:12:28.0437031Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:12:28.0437544Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:12:28.0437925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:12:28.0438269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:12:28.0438515Z dist init r=0, world=2 2022-11-23T01:12:28.0438695Z dist init r=1, world=2 2022-11-23T01:12:28.0438899Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:12:28.0439244Z test_diff_hyperparams_cpu_offload_sharding_strategy_str_no_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups) 2022-11-23T01:12:28.0439716Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8980 2022-11-23T01:12:28.0440113Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8981 2022-11-23T01:12:28.0440565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0441094Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0441599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0441944Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0442263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:12:28.0442726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0443055Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0443480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0443855Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0444186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:12:28.0444680Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:12:28.0445192Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:12:28.0445563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:12:28.0445909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:12:28.0446163Z dist init r=0, world=2 2022-11-23T01:12:28.0446329Z dist init r=1, world=2 2022-11-23T01:12:28.0446533Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:12:28.0446893Z test_diff_hyperparams_cpu_offload_sharding_strategy_str_shard_grad_op (__main__.TestFSDPUseOrigParamsMultipleParamGroups) 2022-11-23T01:12:28.0447360Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9008 2022-11-23T01:12:28.0447753Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9009 2022-11-23T01:12:28.0448202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0448529Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0448945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0449288Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0449621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:12:28.0450087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0450401Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0450826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0451168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0451489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:12:28.0451977Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:12:28.0452493Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:12:28.0452873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:12:28.0453206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:12:28.0453459Z dist init r=0, world=2 2022-11-23T01:12:28.0453639Z dist init r=1, world=2 2022-11-23T01:12:28.0453831Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:12:28.0454217Z test_diff_hyperparams_sharding_strategy_str_full_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups) 2022-11-23T01:12:28.0454688Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9036 2022-11-23T01:12:28.0455083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9037 2022-11-23T01:12:28.0455521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0455849Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0456273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0456648Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0456973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:12:28.0457439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0457763Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0458175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0458515Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0458846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:12:28.0459332Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:12:28.0459839Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:12:28.0460225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:12:28.0460572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:12:28.0460830Z dist init r=0, world=2 2022-11-23T01:12:28.0460995Z dist init r=1, world=2 2022-11-23T01:12:28.0461202Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:12:28.0461545Z test_diff_hyperparams_sharding_strategy_str_no_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups) 2022-11-23T01:12:28.0461998Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9064 2022-11-23T01:12:28.0462391Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9065 2022-11-23T01:12:28.0462846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0463171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0463586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0463928Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0464260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:12:28.0464716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0465037Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0465461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0465804Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0466121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:12:28.0466645Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:12:28.0467169Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:12:28.0467552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:12:28.0467885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:12:28.0468138Z dist init r=0, world=2 2022-11-23T01:12:28.0468314Z dist init r=1, world=2 2022-11-23T01:12:28.0468508Z skip: Need at least 2 CUDA devices (1.715s) 2022-11-23T01:12:28.0468858Z test_diff_hyperparams_sharding_strategy_str_shard_grad_op (__main__.TestFSDPUseOrigParamsMultipleParamGroups) 2022-11-23T01:12:28.0469358Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9092 2022-11-23T01:12:28.0469744Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9093 2022-11-23T01:12:28.0470192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0470517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0470942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0471276Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0471609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:12:28.0472152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0472479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0472894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0473241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0473571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:12:28.0474052Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:12:28.0474570Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:12:28.0474954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:12:28.0475299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:12:28.0475546Z dist init r=1, world=2 2022-11-23T01:12:28.0475727Z dist init r=0, world=2 2022-11-23T01:12:28.0475929Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:12:28.0476227Z test_diff_trainability (__main__.TestFSDPUseOrigParamsMultipleParamGroups) 2022-11-23T01:12:28.0476657Z Tests FSDP parity with DDP when using multiple parameter groups and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9120 2022-11-23T01:12:28.0477144Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9121 2022-11-23T01:12:28.0477593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0477919Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0478342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0478673Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0479006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:12:28.0479509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0479824Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0480245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0480749Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0481087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:12:28.0481570Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:12:28.0482161Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:12:28.0482544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:12:28.0482893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:12:28.0483138Z dist init r=0, world=2 2022-11-23T01:12:28.0483318Z dist init r=1, world=2 2022-11-23T01:12:28.0483528Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:12:28.0483824Z test_multiple_optimizers (__main__.TestFSDPUseOrigParamsMultipleParamGroups) 2022-11-23T01:12:28.0484249Z Tests using two optimizers where only one sets gradients to ``None``. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9148 2022-11-23T01:12:28.0484636Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9149 2022-11-23T01:12:28.0485087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0485403Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0485827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0486174Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0486496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:12:28.0486961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0487287Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0487708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0488038Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0488371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:12:28.0488857Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:12:28.0489380Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:12:28.0489751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:12:28.0490095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:12:28.0490352Z dist init r=0, world=2 2022-11-23T01:12:28.0490517Z dist init r=1, world=2 2022-11-23T01:12:28.0490722Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:12:28.0491017Z test_access_params_after_forward (__main__.TestFSDPUseOrigParamsParamAccess) 2022-11-23T01:12:28.0491423Z Tests that accessing the original parameters after the forward but ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9176 2022-11-23T01:12:28.0491822Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9177 2022-11-23T01:12:28.0492331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0492661Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0493076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0493421Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0493754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:12:28.0494218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0494566Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0494991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0495331Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0495651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:12:28.0496141Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:12:28.0496660Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:12:28.0497042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:12:28.0497372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:12:28.0497628Z dist init r=1, world=2 2022-11-23T01:12:28.0497807Z dist init r=0, world=2 2022-11-23T01:12:28.0498001Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:12:28.0498314Z test_multiple_forward_offload_params_False (__main__.TestFSDPUseOrigParamsUnshardReshard) 2022-11-23T01:12:28.0498742Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9204 2022-11-23T01:12:28.0499121Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9205 2022-11-23T01:12:28.0499558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0499883Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0500311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0500644Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0500981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:12:28.0501441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0501768Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0502181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0502521Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0502852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:12:28.0503336Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:12:28.0503838Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:12:28.0504225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:12:28.0504568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:12:28.0504814Z dist init r=0, world=2 2022-11-23T01:12:28.0505026Z dist init r=1, world=2 2022-11-23T01:12:28.0505235Z skip: Need at least 2 CUDA devices (1.613s) 2022-11-23T01:12:28.0505547Z test_multiple_forward_offload_params_True (__main__.TestFSDPUseOrigParamsUnshardReshard) 2022-11-23T01:12:28.0505961Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9232 2022-11-23T01:12:28.0506343Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9233 2022-11-23T01:12:28.0506795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0507143Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0507569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0507912Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0508248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:12:28.0508697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0509022Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0509443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0509786Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0510106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:12:28.0510600Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:12:28.0511115Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:12:28.0511489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:12:28.0511835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:12:28.0512174Z dist init r=1, world=2 2022-11-23T01:12:28.0512352Z dist init r=0, world=2 2022-11-23T01:12:28.0512546Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:12:28.0512873Z test_summon_between_two_forwards_offload_params_False (__main__.TestFSDPUseOrigParamsUnshardReshard) 2022-11-23T01:12:28.0513307Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9260 2022-11-23T01:12:28.0513680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9261 2022-11-23T01:12:28.0514126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0514458Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0514885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0515215Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0515547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:12:28.0516014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0516323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0516752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0517093Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0517462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:12:28.0517942Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:12:28.0518459Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:12:28.0518838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:12:28.0519181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:12:28.0519428Z dist init r=0, world=2 2022-11-23T01:12:28.0519606Z dist init r=1, world=2 2022-11-23T01:12:28.0519839Z skip: Need at least 2 CUDA devices (1.613s) 2022-11-23T01:12:28.0520153Z test_summon_between_two_forwards_offload_params_True (__main__.TestFSDPUseOrigParamsUnshardReshard) 2022-11-23T01:12:28.0520716Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9288 2022-11-23T01:12:28.0521118Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9289 2022-11-23T01:12:28.0521573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0521889Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0522317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0522659Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0522983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:12:28.0523446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0523770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0524195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0524524Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0524854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:12:28.0525341Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:12:28.0525856Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:12:28.0526231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:12:28.0526574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:12:28.0526830Z dist init r=0, world=2 2022-11-23T01:12:28.0526996Z dist init r=1, world=2 2022-11-23T01:12:28.0527203Z skip: Need at least 2 CUDA devices (1.613s) 2022-11-23T01:12:28.0527476Z test_grad_writeback (__main__.TestFSDPUseOrigParamsWriteback) 2022-11-23T01:12:28.0527973Z Tests that changes to the original parameters' gradients are written ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9316 2022-11-23T01:12:28.0528369Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9317 2022-11-23T01:12:28.0528814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0529143Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0529558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0529901Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0530302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:12:28.0530770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0531084Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0531505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0531847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0532165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:12:28.0532698Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:12:28.0533215Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:12:28.0533599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:12:28.0533928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:12:28.0534179Z dist init r=1, world=2 2022-11-23T01:12:28.0534363Z dist init r=0, world=2 2022-11-23T01:12:28.0534555Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:12:28.0534830Z test_param_writeback (__main__.TestFSDPUseOrigParamsWriteback) 2022-11-23T01:12:28.0535225Z Tests that changes to the original parameters are written back. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9344 2022-11-23T01:12:28.0535616Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9345 2022-11-23T01:12:28.0536052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0536378Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0536808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0537141Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0537474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:12:28.0537940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0538263Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0538673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0539018Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0539346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:12:28.0539837Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:12:28.0540340Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:12:28.0540724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:12:28.0541068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:12:28.0541313Z dist init r=1, world=2 2022-11-23T01:12:28.0541492Z dist init r=0, world=2 2022-11-23T01:12:28.0541699Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:12:28.0542081Z test_writeback_shape_mismatch (__main__.TestFSDPUseOrigParamsWriteback) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9372 2022-11-23T01:12:28.0542485Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9373 2022-11-23T01:12:28.0542967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0543296Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0543709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0544052Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0554142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:12:28.0554781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:12:28.0555206Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:12:28.0555648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:12:28.0556004Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:12:28.0556354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:12:28.0556842Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:12:28.0557367Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:12:28.0557758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:12:28.0558097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:12:28.0558364Z dist init r=0, world=2 2022-11-23T01:12:28.0558548Z dist init r=1, world=2 2022-11-23T01:12:28.0558758Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:12:28.0558881Z 2022-11-23T01:12:28.0559078Z ---------------------------------------------------------------------- 2022-11-23T01:12:28.0559323Z Ran 18 tests in 29.831s 2022-11-23T01:12:28.0559441Z 2022-11-23T01:12:28.0559514Z OK (skipped=18) 2022-11-23T01:12:28.0559622Z 2022-11-23T01:12:28.0559692Z Generating XML reports... 2022-11-23T01:12:28.0560184Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsFQNs-20221123011157.xml 2022-11-23T01:12:28.0561360Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsMultipleParamGroups-20221123011157.xml 2022-11-23T01:12:28.0562302Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsParamAccess-20221123011157.xml 2022-11-23T01:12:28.0562991Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsUnshardReshard-20221123011157.xml 2022-11-23T01:12:28.0563679Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsWriteback-20221123011157.xml 2022-11-23T01:12:28.0563982Z 2022-11-23T01:12:28.0564395Z ##[endgroup] 2022-11-23T01:12:28.0564878Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_use_orig_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_use_orig_params_l6yo7x99) 2022-11-23T01:12:28.0565158Z 2022-11-23T01:12:28.0565365Z Running distributed/optim/test_zero_redundancy_optimizer ... [2022-11-23 01:12:28.034976] 2022-11-23T01:12:28.0565926Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/optim/test_zero_redundancy_optimizer.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:12:28.035313] 2022-11-23T01:13:09.7348660Z 2022-11-23T01:13:09.7349543Z Expand the folded group to see the log file of distributed/optim/test_zero_redundancy_optimizer 2022-11-23T01:13:09.7351629Z ##[group]PRINTING LOG FILE of distributed/optim/test_zero_redundancy_optimizer (/var/lib/jenkins/workspace/test/test-reports/distributed-optim-test_zero_redundancy_optimizer_4awpstll) 2022-11-23T01:13:09.7352845Z Test results will be stored in test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer 2022-11-23T01:13:09.7353242Z 2022-11-23T01:13:09.7353373Z Running tests... 2022-11-23T01:13:09.7353911Z ---------------------------------------------------------------------- 2022-11-23T01:13:09.7385504Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7387047Z Check that ZeroRedundancyOptimizer properly handles adding a new ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/67287 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.262s) 2022-11-23T01:13:09.7388083Z test_collect_shards (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7388594Z Check the state consolidation mechanism and the state dict exposed ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9411 2022-11-23T01:13:09.7388994Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9412 2022-11-23T01:13:09.7389726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7390343Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7390782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7391138Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7391475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:13:09.7392286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7392797Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7393456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7393809Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7394124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:13:09.7394463Z skip: CUDA is not available. (1.818s) 2022-11-23T01:13:09.7394879Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7395392Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:13:09.7396046Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7396889Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:13:09.7397499Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7397991Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:13:09.7398472Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7398985Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:13:09.7400056Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7401371Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:13:09.7402452Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7403514Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:13:09.7404689Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7405751Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:13:09.7406827Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7407885Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:13:09.7408951Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7410015Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:13:09.7411113Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7412182Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:13:09.7413257Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7414300Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:13:09.7415375Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7416437Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:13:09.7417513Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7418560Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:13:09.7419625Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7420684Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:13:09.7421749Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7422806Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:13:09.7423925Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7424990Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:13:09.7425943Z test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7426957Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9433 2022-11-23T01:13:09.7427795Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9434 2022-11-23T01:13:09.7428904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7429625Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7430563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7431326Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7432101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:13:09.7433116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7433810Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7434745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7435500Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7436212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:13:09.7436799Z skip: CUDA is not available. (1.715s) 2022-11-23T01:13:09.7437559Z test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7438588Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9455 2022-11-23T01:13:09.7439407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9456 2022-11-23T01:13:09.7440129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7440687Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7441359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7441917Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7442451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:13:09.7443238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7443746Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7444453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7445029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7445572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:13:09.7446028Z skip: CUDA is not available. (1.716s) 2022-11-23T01:13:09.7446614Z test_local_optimizer_parity_optimizer_class_str_Adam_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7447359Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9477 2022-11-23T01:13:09.7448106Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9478 2022-11-23T01:13:09.7448841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7449375Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7450110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7450737Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7451079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:13:09.7451603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7451936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7452360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7452705Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7453028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:13:09.7453309Z skip: CUDA is not available. (1.716s) 2022-11-23T01:13:09.7453644Z test_local_optimizer_parity_optimizer_class_str_Adam_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7454106Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9499 2022-11-23T01:13:09.7454619Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9500 2022-11-23T01:13:09.7455056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7455387Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7455818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7456164Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7456474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:13:09.7456935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7457261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7457687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7458018Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7458341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:13:09.7458623Z skip: CUDA is not available. (1.716s) 2022-11-23T01:13:09.7458956Z test_local_optimizer_parity_optimizer_class_str_SGD_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7459426Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9521 2022-11-23T01:13:09.7459820Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9522 2022-11-23T01:13:09.7460270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7460587Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7461017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7461363Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7461709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:13:09.7462172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7462501Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7462927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7463262Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7463583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:13:09.7463863Z skip: CUDA is not available. (1.719s) 2022-11-23T01:13:09.7464243Z test_local_optimizer_parity_optimizer_class_str_SGD_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7464691Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9543 2022-11-23T01:13:09.7465085Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9544 2022-11-23T01:13:09.7465536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7465850Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7466279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7466631Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7466954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:13:09.7467402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7467729Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7468157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7468504Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7468812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:13:09.7469094Z skip: CUDA is not available. (1.716s) 2022-11-23T01:13:09.7469385Z test_lr_scheduler (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7469780Z Check that a normal PyTorch ``lr_scheduler`` is usable with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9565 2022-11-23T01:13:09.7470168Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9566 2022-11-23T01:13:09.7470621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7470949Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7471364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7471709Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7472107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:13:09.7472557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7472887Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7473316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7473666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7473974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:13:09.7474256Z skip: CUDA is not available. (1.814s) 2022-11-23T01:13:09.7474591Z test_multiple_param_groups (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7475015Z Check parity between constructing ZeRO with multiple parameter groups ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9587 2022-11-23T01:13:09.7475417Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9588 2022-11-23T01:13:09.7475863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7476192Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7476612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7476991Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7477314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:13:09.7477773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7478088Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7478514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7478862Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7479167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:13:09.7479449Z skip: CUDA is not available. (1.715s) 2022-11-23T01:13:09.7479753Z test_nondefault_process_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7480307Z Check that ZeroRedundancyOptimizer works with a non-default process ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9609 2022-11-23T01:13:09.7480899Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9610 2022-11-23T01:13:09.7481360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7481780Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7482198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7482546Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7482875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:13:09.7483267Z INFO:torch.testing._internal.common_distributed:Skipping `test_nondefault_process_group()` since world size of 2 is less than 4 2022-11-23T01:13:09.7483747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7484080Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7484511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7484862Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7485173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:13:09.7485563Z INFO:torch.testing._internal.common_distributed:Skipping `test_nondefault_process_group()` since world size of 2 is less than 4 2022-11-23T01:13:09.7485846Z ok (1.715s) 2022-11-23T01:13:09.7486094Z test_sharding (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7487060Z Check ZeroRedundancyOptimizer's parameter sharding at construction ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/67295 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s) 2022-11-23T01:13:09.7487699Z test_step (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7488117Z Check that ZeroRedundancyOptimizer properly exposes the ``step()`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9631 2022-11-23T01:13:09.7488506Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9632 2022-11-23T01:13:09.7488965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7489299Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7489731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7490107Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7490428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:13:09.7490931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7491310Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7491724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7492071Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7492393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:13:09.7492659Z skip: CUDA is not available. (1.714s) 2022-11-23T01:13:09.7492959Z test_step_with_closure (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7493383Z Check that ZeroRedundancyOptimizer properly exposes the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9653 2022-11-23T01:13:09.7493782Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9654 2022-11-23T01:13:09.7494221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7494549Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7494976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7495314Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7495639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:13:09.7496102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7496433Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7496846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7497195Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7497517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:13:09.7497789Z skip: CUDA is not available. (1.715s) 2022-11-23T01:13:09.7498076Z test_zero_join_cpu (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7498491Z Check that the ZeRO join hook allows training with uneven inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9675 2022-11-23T01:13:09.7498883Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9676 2022-11-23T01:13:09.7499324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7499650Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7500115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7500461Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7500769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:13:09.7501228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7501556Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7501968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7502315Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7502670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:13:09.7503037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:13:09.7503397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:13:09.7503890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:13:09.7504411Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:13:09.7504819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptjs_art9 2022-11-23T01:13:09.7505197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi_23qkh1 2022-11-23T01:13:09.7505594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptjs_art9/_remote_module_non_scriptable.py 2022-11-23T01:13:09.7505996Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi_23qkh1/_remote_module_non_scriptable.py 2022-11-23T01:13:09.7506363Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T01:13:09.7506723Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T01:13:09.7507205Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up 2022-11-23T01:13:09.7507543Z _warnings.warn(warn_message, ResourceWarning) 2022-11-23T01:13:09.7507970Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up 2022-11-23T01:13:09.7508302Z _warnings.warn(warn_message, ResourceWarning) 2022-11-23T01:13:09.7508508Z ok (1.813s) 2022-11-23T01:13:09.7508755Z test_zero_join_gpu (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7509140Z Check that the ZeRO join hook allows training with uneven inputs ... skip: c10d was not compiled with the NCCL backend (0.001s) 2022-11-23T01:13:09.7509557Z test_zero_model_parallel_parameters_as_bucket_view_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7510116Z Check that ZeRO works with model parallelism where the model's ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9703 2022-11-23T01:13:09.7510498Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9704 2022-11-23T01:13:09.7510951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7511280Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7511713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7512135Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7512463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:13:09.7512927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7513278Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7513709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7514058Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7514380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:13:09.7514660Z skip: Need at least 4 CUDA devices (1.713s) 2022-11-23T01:13:09.7515001Z test_zero_model_parallel_parameters_as_bucket_view_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T01:13:09.7515554Z Check that ZeRO works with model parallelism where the model's ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9725 2022-11-23T01:13:09.7515965Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9726 2022-11-23T01:13:09.7516418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7516750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7517180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7517512Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7517838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:13:09.7518301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7518629Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7519043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7519389Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7519713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:13:09.7519993Z skip: Need at least 4 CUDA devices (1.713s) 2022-11-23T01:13:09.7520285Z test_constructor (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-11-23T01:13:09.7520856Z Check the robustness of the ZeroRedundancyOptimizer constructor by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9747 2022-11-23T01:13:09.7521395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7521711Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7522138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7522484Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7522797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:13:09.7523160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:13:09.7523657Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T01:13:09.7523946Z ok (1.710s) 2022-11-23T01:13:09.7524195Z test_lr_scheduler (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-11-23T01:13:09.7524598Z Check that a normal PyTorch ``lr_scheduler`` is usable with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9761 2022-11-23T01:13:09.7525102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7525435Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7525850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7526250Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7526574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:13:09.7526929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:13:09.7527425Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T01:13:09.7527715Z ok (1.709s) 2022-11-23T01:13:09.7527983Z test_same_dense_param_type (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-11-23T01:13:09.7528407Z Check that ZeroRedundancyOptimizer raises an exception if the input ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9775 2022-11-23T01:13:09.7528972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7529307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7529724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7530070Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7530395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:13:09.7530756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:13:09.7531238Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T01:13:09.7531531Z ok (1.709s) 2022-11-23T01:13:09.7531786Z test_state_dict (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-11-23T01:13:09.7532198Z Check that ZeroRedundancyOptimizer exposes the expected state dict ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9789 2022-11-23T01:13:09.7532727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7533055Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7533483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7533812Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7534135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:13:09.7534500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:13:09.7534994Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T01:13:09.7535273Z ok (1.711s) 2022-11-23T01:13:09.7535542Z test_step_with_extra_inner_key (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-11-23T01:13:09.7535980Z Check that ZeroRedundancyOptimizer wrapping an optimizer that adds ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9803 2022-11-23T01:13:09.7536499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7536833Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7537261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7537609Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7537922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:13:09.7538285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:13:09.7538815Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T01:13:09.7539106Z ok (1.609s) 2022-11-23T01:13:09.7539357Z test_step_with_kwargs (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-11-23T01:13:09.7539775Z Check that the ``step(**kwargs)`` interface is properly exposed. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9817 2022-11-23T01:13:09.7540287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7540604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7541033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7541411Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7541736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:13:09.7542085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:13:09.7542579Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T01:13:09.7542867Z ok (1.709s) 2022-11-23T01:13:09.7543121Z test_step_without_closure (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-11-23T01:13:09.7543537Z Check that the ``step()`` method (without closure) is handled as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9831 2022-11-23T01:13:09.7544049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7544383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7544801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7545145Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7545470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:13:09.7545821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:13:09.7546316Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T01:13:09.7546604Z ok (1.709s) 2022-11-23T01:13:09.7546857Z test_zero_grad (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-11-23T01:13:09.7547246Z Check that the ``zero_grad`` method is properly handled. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9845 2022-11-23T01:13:09.7547753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:13:09.7548082Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:13:09.7548514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:13:09.7548845Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:13:09.7549168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:13:09.7549531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:13:09.7550011Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T01:13:09.7550302Z ok (1.709s) 2022-11-23T01:13:09.7550407Z 2022-11-23T01:13:09.7550608Z ---------------------------------------------------------------------- 2022-11-23T01:13:09.7550855Z Ran 42 tests in 39.887s 2022-11-23T01:13:09.7550958Z 2022-11-23T01:13:09.7551031Z OK (skipped=32) 2022-11-23T01:13:09.7551140Z 2022-11-23T01:13:09.7551224Z Generating XML reports... 2022-11-23T01:13:09.7551809Z Generated XML report: test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerDistributed-20221123011229.xml 2022-11-23T01:13:09.7552644Z Generated XML report: test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerSingleRank-20221123011229.xml 2022-11-23T01:13:09.7552976Z 2022-11-23T01:13:09.7553302Z ##[endgroup] 2022-11-23T01:13:09.7553811Z FINISHED PRINTING LOG FILE of distributed/optim/test_zero_redundancy_optimizer (/var/lib/jenkins/workspace/test/test-reports/distributed-optim-test_zero_redundancy_optimizer_4awpstll) 2022-11-23T01:13:09.7554105Z 2022-11-23T01:13:09.7554307Z Running distributed/_tensor/test_device_mesh ... [2022-11-23 01:13:09.735689] 2022-11-23T01:13:09.7554859Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_device_mesh.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:13:09.736028] 2022-11-23T01:14:08.0746032Z 2022-11-23T01:14:08.0746491Z Expand the folded group to see the log file of distributed/_tensor/test_device_mesh 2022-11-23T01:14:08.0747453Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_device_mesh (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_device_mesh_ogec3cgs) 2022-11-23T01:14:08.0754695Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_device_mesh 2022-11-23T01:14:08.0755067Z 2022-11-23T01:14:08.0755180Z Running tests... 2022-11-23T01:14:08.0755660Z ---------------------------------------------------------------------- 2022-11-23T01:14:08.0756293Z test_all_gather_1d (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9870 2022-11-23T01:14:08.0756943Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9871 2022-11-23T01:14:08.0791932Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9872 2022-11-23T01:14:08.0792502Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9873 2022-11-23T01:14:08.0793036Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 9874 2022-11-23T01:14:08.0793721Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 9875 2022-11-23T01:14:08.0794252Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 9876 2022-11-23T01:14:08.0794757Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 9877 2022-11-23T01:14:08.0795549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0796127Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0796822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0797348Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0797844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:14:08.0798600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0799138Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0799847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0800397Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0801083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T01:14:08.0801885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0802424Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0803009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0803414Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0803957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T01:14:08.0804746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0805293Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0805998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0806557Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0807149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T01:14:08.0807910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0808447Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0809112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0809671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0810207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:14:08.0810981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0811496Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0812198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0812748Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0813278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:14:08.0814042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0814585Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0815288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0815838Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0816329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:14:08.0817070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0817591Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0818258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0818811Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0819354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T01:14:08.0820016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:14:08.0820428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:14:08.0820793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:14:08.0821153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:14:08.0821516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4 2022-11-23T01:14:08.0821861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5 2022-11-23T01:14:08.0822267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7 2022-11-23T01:14:08.0822764Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.0823273Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.0823663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6 2022-11-23T01:14:08.0824150Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.0824696Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.0825192Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.0825702Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.0826211Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.0826722Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.0826993Z ok (3.089s) 2022-11-23T01:14:08.0827321Z test_all_gather_nd (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9982 2022-11-23T01:14:08.0827711Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9983 2022-11-23T01:14:08.0828037Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9984 2022-11-23T01:14:08.0828344Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9985 2022-11-23T01:14:08.0828661Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 9986 2022-11-23T01:14:08.0828978Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 9987 2022-11-23T01:14:08.0829283Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 9988 2022-11-23T01:14:08.0829594Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 9989 2022-11-23T01:14:08.0830040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0830368Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0830781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0831131Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0831449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:14:08.0833517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0833854Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0834280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0834624Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0834930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:14:08.0835389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0835718Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0836130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0836469Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0836869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T01:14:08.0837332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0837648Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0838072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0838414Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0838737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T01:14:08.0839214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0839540Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0839964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0840293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0840735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:14:08.0841202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0841526Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0841936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0842278Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0842596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T01:14:08.0843037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0843369Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0843790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0844132Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0844437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T01:14:08.0844892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0845216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0845639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0845966Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0846286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:14:08.0846647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:14:08.0846998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:14:08.0847358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:14:08.0847719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:14:08.0848070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4 2022-11-23T01:14:08.0848413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5 2022-11-23T01:14:08.0848765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7 2022-11-23T01:14:08.0849203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6 2022-11-23T01:14:08.0849696Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.0850204Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.0850715Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.0851226Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.0851788Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.0852282Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.0852792Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.0853302Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.0853689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T01:14:08.0854036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 6 2022-11-23T01:14:08.0854397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-11-23T01:14:08.0854755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2022-11-23T01:14:08.0855097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5 2022-11-23T01:14:08.0855450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 7 2022-11-23T01:14:08.0855803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T01:14:08.0856153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4 2022-11-23T01:14:08.0856625Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.0857137Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.0857648Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.0858153Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.0858647Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.0859151Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.0859658Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.0860162Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.0860536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T01:14:08.0860897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T01:14:08.0861256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 5 2022-11-23T01:14:08.0861601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3 2022-11-23T01:14:08.0861990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 7 2022-11-23T01:14:08.0862347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 4 2022-11-23T01:14:08.0862699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2 2022-11-23T01:14:08.0863038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 6 2022-11-23T01:14:08.0863518Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.0864028Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.0864570Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.0865069Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.0865572Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.0865962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T01:14:08.0866442Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.0866815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2 2022-11-23T01:14:08.0867169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3 2022-11-23T01:14:08.0867536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 7 2022-11-23T01:14:08.0867886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 6 2022-11-23T01:14:08.0868371Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.0868880Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.0869271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 4 2022-11-23T01:14:08.0869613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T01:14:08.0869967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 5 2022-11-23T01:14:08.0870455Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.0870961Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.0871454Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.0872049Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.0872440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2 2022-11-23T01:14:08.0872800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 6 2022-11-23T01:14:08.0873147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1 2022-11-23T01:14:08.0873506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 5 2022-11-23T01:14:08.0873989Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.0874410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 4 2022-11-23T01:14:08.0874885Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.0875389Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.0875868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0 2022-11-23T01:14:08.0876342Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.0876788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 7 2022-11-23T01:14:08.0877145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3 2022-11-23T01:14:08.0877632Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.0878126Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.0878633Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.0879145Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.0879654Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.0880147Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.0880535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 3 2022-11-23T01:14:08.0881033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1 2022-11-23T01:14:08.0881397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 6 2022-11-23T01:14:08.0881742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 5 2022-11-23T01:14:08.0882233Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.0882622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 4 2022-11-23T01:14:08.0883107Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.0883486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 7 2022-11-23T01:14:08.0883842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 2 2022-11-23T01:14:08.0884202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0 2022-11-23T01:14:08.0884673Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.0885179Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.0885682Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.0886188Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.0886695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 5 2022-11-23T01:14:08.0887055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0 2022-11-23T01:14:08.0887481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 6 2022-11-23T01:14:08.0887845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 2 2022-11-23T01:14:08.0888317Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.0888705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 4 2022-11-23T01:14:08.0889186Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.0889700Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.0890118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 7 2022-11-23T01:14:08.0890605Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.0890996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 3 2022-11-23T01:14:08.0891358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1 2022-11-23T01:14:08.0891828Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.0892347Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.0892855Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.0893362Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.0893860Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.0894249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0 2022-11-23T01:14:08.0894732Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.0895108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 5 2022-11-23T01:14:08.0895464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1 2022-11-23T01:14:08.0895823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 2 2022-11-23T01:14:08.0896311Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.0896803Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.0897192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 7 2022-11-23T01:14:08.0897555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 3 2022-11-23T01:14:08.0897911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 6 2022-11-23T01:14:08.0898254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 4 2022-11-23T01:14:08.0898738Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.0899244Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.0899630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 4 2022-11-23T01:14:08.0900130Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.0900637Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.0901139Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.0901639Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.0902130Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.0902667Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.0903056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 3 2022-11-23T01:14:08.0903416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1 2022-11-23T01:14:08.0903757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0 2022-11-23T01:14:08.0904114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 2 2022-11-23T01:14:08.0904466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 6 2022-11-23T01:14:08.0904804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 7 2022-11-23T01:14:08.0905157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 5 2022-11-23T01:14:08.0905641Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.0906147Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.0906643Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.0907032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 5 2022-11-23T01:14:08.0907400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 2 2022-11-23T01:14:08.0907886Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.0909661Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.0910055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 4 2022-11-23T01:14:08.0910420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 6 2022-11-23T01:14:08.0910907Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.0911401Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.0911996Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.0912392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 3 2022-11-23T01:14:08.0912758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 7 2022-11-23T01:14:08.0913112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1 2022-11-23T01:14:08.0913477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0 2022-11-23T01:14:08.0914021Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.0914521Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.0915029Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.0915538Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.0915930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 7 2022-11-23T01:14:08.0916317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0 2022-11-23T01:14:08.0916807Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.0917321Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.0917711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 4 2022-11-23T01:14:08.0918060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 6 2022-11-23T01:14:08.0918546Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.0919046Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.0919437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 5 2022-11-23T01:14:08.0919798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1 2022-11-23T01:14:08.0920160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 3 2022-11-23T01:14:08.0920522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 2 2022-11-23T01:14:08.0921158Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.0921660Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.0922160Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.0922554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1 2022-11-23T01:14:08.0922908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 2 2022-11-23T01:14:08.0923392Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.0923902Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.0924293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 6 2022-11-23T01:14:08.0924764Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.0925272Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.0925780Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.0926170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 7 2022-11-23T01:14:08.0926523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0 2022-11-23T01:14:08.0926950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 3 2022-11-23T01:14:08.0927317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 4 2022-11-23T01:14:08.0927683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 5 2022-11-23T01:14:08.0928159Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.0928671Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.0929216Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.0929725Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.0930221Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.0930612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1 2022-11-23T01:14:08.0930975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 2 2022-11-23T01:14:08.0931458Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.0931835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 4 2022-11-23T01:14:08.0932201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0 2022-11-23T01:14:08.0932564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 3 2022-11-23T01:14:08.0932915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 5 2022-11-23T01:14:08.0933403Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.0933914Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.0934301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 6 2022-11-23T01:14:08.0934774Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.0935162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 7 2022-11-23T01:14:08.0935650Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.0936168Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.0936664Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.0937172Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.0937670Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.0938180Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.0938677Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.0938962Z ok (3.238s) 2022-11-23T01:14:08.0939292Z test_all_gather_uneven (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10166 2022-11-23T01:14:08.0939738Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10167 2022-11-23T01:14:08.0940058Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10168 2022-11-23T01:14:08.0940380Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10169 2022-11-23T01:14:08.0940708Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 10170 2022-11-23T01:14:08.0941018Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 10171 2022-11-23T01:14:08.0941336Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 10172 2022-11-23T01:14:08.0941687Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 10173 2022-11-23T01:14:08.0942143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0942465Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0942895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0943244Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0943554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T01:14:08.0944014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0944340Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0944769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0945100Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0945423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:14:08.0945885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0946209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0946623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0946964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0947284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:14:08.0947732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0948059Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0948483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0948827Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0949139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T01:14:08.0949594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0949916Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0950328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0950671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0950991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:14:08.0951449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0951759Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0952295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0952640Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0952958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:14:08.0953400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0953722Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0954150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0954515Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0954834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T01:14:08.0955297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0955623Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0956038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0956380Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0956703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T01:14:08.0957053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:14:08.0957418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:14:08.0957782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:14:08.0958139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:14:08.0958481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4 2022-11-23T01:14:08.0958828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6 2022-11-23T01:14:08.0959180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5 2022-11-23T01:14:08.0959532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7 2022-11-23T01:14:08.0960013Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.0960532Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.0961392Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.0961910Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.0962403Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.0962916Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.0963424Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.0963920Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.0964204Z ok (2.739s) 2022-11-23T01:14:08.0964527Z test_all_reduce_1d (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10278 2022-11-23T01:14:08.0964921Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10279 2022-11-23T01:14:08.0965296Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10280 2022-11-23T01:14:08.0965621Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10281 2022-11-23T01:14:08.0965943Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 10282 2022-11-23T01:14:08.0966262Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 10283 2022-11-23T01:14:08.0966576Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 10284 2022-11-23T01:14:08.0966896Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 10285 2022-11-23T01:14:08.0967392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0967709Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0968137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0968482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0968803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T01:14:08.0969250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0969579Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0970003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0970335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0970659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T01:14:08.0971112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0971441Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0971859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0972200Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0972519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:14:08.0972965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0973289Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0973718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0974060Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0974368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T01:14:08.0974822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0975146Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0975567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0975895Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0976214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:14:08.0976667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0976980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0977407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0977784Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0978107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T01:14:08.0978553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0978875Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0979301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0979631Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0979980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:14:08.0980435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0980759Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0981171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0981509Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0981824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:14:08.0982185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:14:08.0982539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:14:08.0982898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:14:08.0983259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5 2022-11-23T01:14:08.0983600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:14:08.0983955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4 2022-11-23T01:14:08.0984309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6 2022-11-23T01:14:08.0984658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7 2022-11-23T01:14:08.0985136Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.0985655Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.0986170Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.0986678Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.0987176Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.0987687Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.0988198Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.0988709Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.0988982Z ok (2.839s) 2022-11-23T01:14:08.0989303Z test_all_reduce_nd (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10390 2022-11-23T01:14:08.0989695Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10391 2022-11-23T01:14:08.0990014Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10392 2022-11-23T01:14:08.0990369Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10393 2022-11-23T01:14:08.0990689Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 10394 2022-11-23T01:14:08.0991007Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 10395 2022-11-23T01:14:08.0991314Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 10396 2022-11-23T01:14:08.0991629Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 10397 2022-11-23T01:14:08.0992165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0992524Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0992952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0993297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0993618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:14:08.0994064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0994387Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0994809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0995158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0995467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:14:08.0995927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0996249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0996664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0997006Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0997329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:14:08.0997785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0998098Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.0998524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.0998868Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.0999173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T01:14:08.0999631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.0999956Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1000379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1001036Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1001363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T01:14:08.1001820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1002147Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1002556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1002898Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1003277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T01:14:08.1003723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1004046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1004471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1004810Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1005114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:14:08.1005612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1005938Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1006351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1006692Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1007012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T01:14:08.1007376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:14:08.1007727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:14:08.1008086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:14:08.1008449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:14:08.1008810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4 2022-11-23T01:14:08.1009162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5 2022-11-23T01:14:08.1009522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6 2022-11-23T01:14:08.1010009Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1010393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7 2022-11-23T01:14:08.1010878Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1011488Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1012001Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1012500Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1013007Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1013520Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1014033Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1014406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T01:14:08.1014767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-11-23T01:14:08.1015127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5 2022-11-23T01:14:08.1015490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 6 2022-11-23T01:14:08.1015864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 7 2022-11-23T01:14:08.1016218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2022-11-23T01:14:08.1016616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4 2022-11-23T01:14:08.1016958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T01:14:08.1017444Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1017832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T01:14:08.1018350Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1018852Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1019359Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1019862Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1020374Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1020749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 5 2022-11-23T01:14:08.1021231Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1021620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 7 2022-11-23T01:14:08.1021981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3 2022-11-23T01:14:08.1022327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 4 2022-11-23T01:14:08.1022811Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1023205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T01:14:08.1023565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 6 2022-11-23T01:14:08.1023908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2 2022-11-23T01:14:08.1024393Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1024905Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1025418Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1025909Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1026416Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1026807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 7 2022-11-23T01:14:08.1027150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3 2022-11-23T01:14:08.1027513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 4 2022-11-23T01:14:08.1027871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2 2022-11-23T01:14:08.1028394Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1028900Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1029289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T01:14:08.1029771Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1030156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 6 2022-11-23T01:14:08.1030500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 5 2022-11-23T01:14:08.1030885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T01:14:08.1031371Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1031960Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1032461Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1032971Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1033477Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1033982Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1034479Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1034870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2 2022-11-23T01:14:08.1035230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1 2022-11-23T01:14:08.1035576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 4 2022-11-23T01:14:08.1035931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 6 2022-11-23T01:14:08.1036413Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1036801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0 2022-11-23T01:14:08.1037148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 5 2022-11-23T01:14:08.1037500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 7 2022-11-23T01:14:08.1037856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3 2022-11-23T01:14:08.1038336Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1038834Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1039345Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1039850Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1040354Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1040944Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1041396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 3 2022-11-23T01:14:08.1041756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 4 2022-11-23T01:14:08.1042114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1 2022-11-23T01:14:08.1042459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 6 2022-11-23T01:14:08.1042941Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1043452Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1043865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 5 2022-11-23T01:14:08.1044222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 7 2022-11-23T01:14:08.1044579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0 2022-11-23T01:14:08.1044933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 2 2022-11-23T01:14:08.1045398Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1045906Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1046291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0 2022-11-23T01:14:08.1046651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 2 2022-11-23T01:14:08.1047120Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1047514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 5 2022-11-23T01:14:08.1047998Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1048502Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1048997Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1049499Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1050011Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1050394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 7 2022-11-23T01:14:08.1050740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 4 2022-11-23T01:14:08.1051099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 6 2022-11-23T01:14:08.1051458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 3 2022-11-23T01:14:08.1051800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1 2022-11-23T01:14:08.1052282Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1052789Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1053295Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1053703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1 2022-11-23T01:14:08.1054185Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1054685Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1055072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0 2022-11-23T01:14:08.1055420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 2 2022-11-23T01:14:08.1055901Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1056323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 5 2022-11-23T01:14:08.1056803Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1057296Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1057686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 7 2022-11-23T01:14:08.1058048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 3 2022-11-23T01:14:08.1058407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 4 2022-11-23T01:14:08.1058746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 6 2022-11-23T01:14:08.1059233Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1059745Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1060243Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1060629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 4 2022-11-23T01:14:08.1061110Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1061491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 6 2022-11-23T01:14:08.1061956Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1062346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1 2022-11-23T01:14:08.1062705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 3 2022-11-23T01:14:08.1063191Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1063682Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1064069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0 2022-11-23T01:14:08.1064426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 2 2022-11-23T01:14:08.1064905Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1065283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 5 2022-11-23T01:14:08.1065636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 7 2022-11-23T01:14:08.1066147Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1066653Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1067033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 5 2022-11-23T01:14:08.1067402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 7 2022-11-23T01:14:08.1086262Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1086901Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1087425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 4 2022-11-23T01:14:08.1087932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1088440Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1088830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 6 2022-11-23T01:14:08.1089204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 3 2022-11-23T01:14:08.1089683Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1090201Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1090606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 2 2022-11-23T01:14:08.1090980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0 2022-11-23T01:14:08.1091338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1 2022-11-23T01:14:08.1091832Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1092353Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1092750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0 2022-11-23T01:14:08.1093101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1 2022-11-23T01:14:08.1093595Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1094112Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1094631Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1095012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 5 2022-11-23T01:14:08.1095376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 7 2022-11-23T01:14:08.1095869Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1096385Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1096769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 4 2022-11-23T01:14:08.1097257Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1097697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 6 2022-11-23T01:14:08.1098052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 2 2022-11-23T01:14:08.1098417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 3 2022-11-23T01:14:08.1098908Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1099423Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1099804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 2 2022-11-23T01:14:08.1100199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 3 2022-11-23T01:14:08.1100689Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1101202Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1101583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 7 2022-11-23T01:14:08.1102072Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1102583Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1102980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 6 2022-11-23T01:14:08.1103454Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1103971Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1104362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0 2022-11-23T01:14:08.1104728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1 2022-11-23T01:14:08.1105078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 5 2022-11-23T01:14:08.1105441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 4 2022-11-23T01:14:08.1105932Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1106431Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1106945Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1107340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 4 2022-11-23T01:14:08.1107824Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1108318Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1108707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0 2022-11-23T01:14:08.1109075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1 2022-11-23T01:14:08.1109562Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1110062Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1110483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 2 2022-11-23T01:14:08.1110848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 3 2022-11-23T01:14:08.1111335Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1111716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 5 2022-11-23T01:14:08.1112183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 7 2022-11-23T01:14:08.1112550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 6 2022-11-23T01:14:08.1113073Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1113568Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1114075Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1114586Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1115096Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1115598Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1116114Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1116626Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1116918Z ok (3.239s) 2022-11-23T01:14:08.1117230Z test_all_to_all_1d (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10574 2022-11-23T01:14:08.1117627Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10575 2022-11-23T01:14:08.1117960Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10576 2022-11-23T01:14:08.1118281Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10577 2022-11-23T01:14:08.1118610Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 10578 2022-11-23T01:14:08.1118938Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 10579 2022-11-23T01:14:08.1119269Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 10580 2022-11-23T01:14:08.1119584Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 10581 2022-11-23T01:14:08.1120039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1120376Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1120916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1121264Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1121591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T01:14:08.1122052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1122373Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1122807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1123155Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1123534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T01:14:08.1123999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1124332Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1124763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1125093Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1125418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:14:08.1125917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1126247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1126664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1127010Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1127333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T01:14:08.1127777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1128105Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1128533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1128881Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1129187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:14:08.1129646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1129973Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1130388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1130735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1131061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:14:08.1131515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1131827Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1132258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1132605Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1132915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:14:08.1133373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1133704Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1134133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1134461Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1134784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T01:14:08.1135148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:14:08.1135522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:14:08.1135913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:14:08.1136283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:14:08.1136650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4 2022-11-23T01:14:08.1137002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5 2022-11-23T01:14:08.1137368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7 2022-11-23T01:14:08.1137734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6 2022-11-23T01:14:08.1138260Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1138770Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1139292Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1139811Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1140328Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1140829Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1141343Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1141861Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1142150Z ok (2.938s) 2022-11-23T01:14:08.1142463Z test_all_to_all_nd (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10686 2022-11-23T01:14:08.1142858Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10687 2022-11-23T01:14:08.1143193Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10688 2022-11-23T01:14:08.1143509Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10689 2022-11-23T01:14:08.1143841Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 10690 2022-11-23T01:14:08.1144169Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 10691 2022-11-23T01:14:08.1144499Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 10692 2022-11-23T01:14:08.1144817Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 10693 2022-11-23T01:14:08.1145267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1145605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1146025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1146374Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1146700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T01:14:08.1147164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1147481Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1147910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1148257Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1148612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:14:08.1149058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1149386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1149814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1150148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1150470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T01:14:08.1150931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1151292Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1151707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1152133Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1152458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:14:08.1152908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1153237Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1153663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1154011Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1154324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:14:08.1154786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1155121Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1155547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1155877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1156197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:14:08.1156653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1156966Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1157390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1157735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1158057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T01:14:08.1158499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1158832Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1159261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1159593Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1159916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T01:14:08.1160282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:14:08.1160768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:14:08.1161125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:14:08.1161550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:14:08.1161918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6 2022-11-23T01:14:08.1162283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4 2022-11-23T01:14:08.1162632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5 2022-11-23T01:14:08.1162992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7 2022-11-23T01:14:08.1163504Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1164336Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1165153Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1166029Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1166754Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1167517Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1168299Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1168852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5 2022-11-23T01:14:08.1169444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 7 2022-11-23T01:14:08.1170265Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1170929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T01:14:08.1171534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-11-23T01:14:08.1172139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 6 2022-11-23T01:14:08.1172716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2022-11-23T01:14:08.1173259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T01:14:08.1173879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4 2022-11-23T01:14:08.1174765Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1175664Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1176315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T01:14:08.1176868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 4 2022-11-23T01:14:08.1177702Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1178615Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1179493Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1180223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T01:14:08.1181246Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1182214Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1182933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3 2022-11-23T01:14:08.1183607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 5 2022-11-23T01:14:08.1184521Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1185253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 7 2022-11-23T01:14:08.1185952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2 2022-11-23T01:14:08.1186622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 6 2022-11-23T01:14:08.1187536Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1188484Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1189434Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1190153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2 2022-11-23T01:14:08.1190826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3 2022-11-23T01:14:08.1191480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 6 2022-11-23T01:14:08.1192467Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1193423Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1194388Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1195218Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1195902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T01:14:08.1196562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 4 2022-11-23T01:14:08.1197237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 7 2022-11-23T01:14:08.1198275Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1199013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T01:14:08.1199678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 5 2022-11-23T01:14:08.1200718Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1201678Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1202638Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1203600Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1204332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1 2022-11-23T01:14:08.1205294Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1206016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0 2022-11-23T01:14:08.1206681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 5 2022-11-23T01:14:08.1207355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 4 2022-11-23T01:14:08.1208252Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1209198Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1210202Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1210882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2 2022-11-23T01:14:08.1211546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 6 2022-11-23T01:14:08.1212218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3 2022-11-23T01:14:08.1212885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 7 2022-11-23T01:14:08.1213801Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1214749Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1215475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 3 2022-11-23T01:14:08.1216140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 7 2022-11-23T01:14:08.1217035Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1217990Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1218714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 6 2022-11-23T01:14:08.1219610Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1220545Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1221498Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1222420Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1223156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1 2022-11-23T01:14:08.1223823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 5 2022-11-23T01:14:08.1224394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 4 2022-11-23T01:14:08.1224929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0 2022-11-23T01:14:08.1225436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 2 2022-11-23T01:14:08.1226187Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1227139Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1228056Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1228912Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1229497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0 2022-11-23T01:14:08.1230014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 4 2022-11-23T01:14:08.1230555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 2 2022-11-23T01:14:08.1231412Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1231959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 7 2022-11-23T01:14:08.1232449Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1232849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 6 2022-11-23T01:14:08.1233336Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1233858Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1234242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 5 2022-11-23T01:14:08.1234614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1 2022-11-23T01:14:08.1234986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 3 2022-11-23T01:14:08.1235461Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1235978Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1236493Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1236886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1 2022-11-23T01:14:08.1237236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 5 2022-11-23T01:14:08.1237725Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1238119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 3 2022-11-23T01:14:08.1238609Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1239110Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1239505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0 2022-11-23T01:14:08.1239991Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1240388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 2 2022-11-23T01:14:08.1240864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 7 2022-11-23T01:14:08.1241430Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1241908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 4 2022-11-23T01:14:08.1242304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 6 2022-11-23T01:14:08.1242893Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1243413Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1243809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 4 2022-11-23T01:14:08.1244158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 6 2022-11-23T01:14:08.1244649Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1245203Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1245724Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1246100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1 2022-11-23T01:14:08.1246465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 3 2022-11-23T01:14:08.1246952Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1247463Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1247962Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1248361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0 2022-11-23T01:14:08.1248727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 2 2022-11-23T01:14:08.1249098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 7 2022-11-23T01:14:08.1249448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 5 2022-11-23T01:14:08.1249934Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1250447Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1250842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 5 2022-11-23T01:14:08.1251317Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1251830Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1252344Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1252726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 4 2022-11-23T01:14:08.1253216Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1253609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 6 2022-11-23T01:14:08.1253975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 3 2022-11-23T01:14:08.1254450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1254968Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1255403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 2 2022-11-23T01:14:08.1255773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 7 2022-11-23T01:14:08.1256118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1 2022-11-23T01:14:08.1256482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0 2022-11-23T01:14:08.1256976Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1257495Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1258031Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1258547Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1258945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 5 2022-11-23T01:14:08.1259310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 7 2022-11-23T01:14:08.1259658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0 2022-11-23T01:14:08.1260148Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1260658Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1261039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 4 2022-11-23T01:14:08.1261401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 6 2022-11-23T01:14:08.1261893Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1262408Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1262788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1 2022-11-23T01:14:08.1263150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 2 2022-11-23T01:14:08.1263510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 3 2022-11-23T01:14:08.1263997Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1264499Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1265011Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1265523Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1266035Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1266417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 7 2022-11-23T01:14:08.1266787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 2 2022-11-23T01:14:08.1267152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 3 2022-11-23T01:14:08.1267512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0 2022-11-23T01:14:08.1268019Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1268535Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1268930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 6 2022-11-23T01:14:08.1269423Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1269800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1 2022-11-23T01:14:08.1270165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 4 2022-11-23T01:14:08.1270554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 5 2022-11-23T01:14:08.1271026Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1271544Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1272138Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1272661Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1273044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 3 2022-11-23T01:14:08.1273409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 4 2022-11-23T01:14:08.1273777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 5 2022-11-23T01:14:08.1274140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0 2022-11-23T01:14:08.1274616Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1275131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1275522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1 2022-11-23T01:14:08.1276011Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1276508Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1276898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 2 2022-11-23T01:14:08.1277262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 6 2022-11-23T01:14:08.1277751Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1278125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 7 2022-11-23T01:14:08.1278610Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1279124Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1279623Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1280137Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1280764Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1281344Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1281843Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1282132Z ok (3.139s) 2022-11-23T01:14:08.1282465Z test_broadcast_1d (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10870 2022-11-23T01:14:08.1282864Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10871 2022-11-23T01:14:08.1283185Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10872 2022-11-23T01:14:08.1283554Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10873 2022-11-23T01:14:08.1283884Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 10874 2022-11-23T01:14:08.1284200Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 10875 2022-11-23T01:14:08.1284528Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 10876 2022-11-23T01:14:08.1284854Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 10877 2022-11-23T01:14:08.1285310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1285630Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1286061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1286412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1286737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T01:14:08.1287185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1287516Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1287947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1288279Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1288605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:14:08.1289065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1289393Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1289812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1290158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1290481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:14:08.1290927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1291259Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1291686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1292028Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1292335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:14:08.1292792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1293124Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1293555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1293919Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1294243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T01:14:08.1294707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1295017Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1295442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1295784Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1296138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T01:14:08.1296581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1296907Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1297333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1297661Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1297983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:14:08.1298442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1298771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1299185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1299532Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1299856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T01:14:08.1300223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:14:08.1300581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:14:08.1300952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:14:08.1301323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:14:08.1301672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4 2022-11-23T01:14:08.1302036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5 2022-11-23T01:14:08.1302401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7 2022-11-23T01:14:08.1302892Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1303276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6 2022-11-23T01:14:08.1303762Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1304272Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1304793Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1305285Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1305799Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1306313Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1306860Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1307135Z ok (2.838s) 2022-11-23T01:14:08.1307461Z test_broadcast_nd (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10982 2022-11-23T01:14:08.1307858Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10983 2022-11-23T01:14:08.1308176Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10984 2022-11-23T01:14:08.1308508Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10985 2022-11-23T01:14:08.1308861Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 10986 2022-11-23T01:14:08.1309191Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 10987 2022-11-23T01:14:08.1309507Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 10988 2022-11-23T01:14:08.1309832Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 10989 2022-11-23T01:14:08.1310285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1310605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1311027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1311376Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1311703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:14:08.1312239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1312568Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1313002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1313351Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1313660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T01:14:08.1314119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1314450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1314861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1315209Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1315534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:14:08.1315996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1316312Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1316736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1317078Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1317389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T01:14:08.1317846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1318176Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1318603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1318933Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1319295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:14:08.1319752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1320074Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1320484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1320936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1321259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T01:14:08.1321757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1322085Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1322515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1322859Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1323163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:14:08.1323614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1323937Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1324357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1324700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1325018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T01:14:08.1325381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:14:08.1325897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:14:08.1326575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:14:08.1326983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:14:08.1327344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4 2022-11-23T01:14:08.1327694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5 2022-11-23T01:14:08.1328053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6 2022-11-23T01:14:08.1328411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7 2022-11-23T01:14:08.1328896Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1329416Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1329925Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1330313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T01:14:08.1330781Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1331291Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1331804Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1332389Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1332888Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1333281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5 2022-11-23T01:14:08.1333643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-11-23T01:14:08.1334001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2022-11-23T01:14:08.1334348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 6 2022-11-23T01:14:08.1334742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 7 2022-11-23T01:14:08.1335099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4 2022-11-23T01:14:08.1335444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T01:14:08.1335930Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1336440Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1336825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T01:14:08.1337293Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1337680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 5 2022-11-23T01:14:08.1338166Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1338675Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1339173Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1339555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3 2022-11-23T01:14:08.1340034Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1340413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 7 2022-11-23T01:14:08.1340876Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1341262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T01:14:08.1341626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 4 2022-11-23T01:14:08.1341986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2 2022-11-23T01:14:08.1342455Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1342837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 6 2022-11-23T01:14:08.1343319Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1343822Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1344202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2 2022-11-23T01:14:08.1344557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 7 2022-11-23T01:14:08.1344947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 6 2022-11-23T01:14:08.1345417Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1345925Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1346435Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1346823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T01:14:08.1347203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 4 2022-11-23T01:14:08.1347690Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1348204Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1348589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3 2022-11-23T01:14:08.1348938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T01:14:08.1349303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 5 2022-11-23T01:14:08.1349784Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1350291Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1350788Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1351177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1 2022-11-23T01:14:08.1351658Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1352118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 5 2022-11-23T01:14:08.1352468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0 2022-11-23T01:14:08.1352828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 4 2022-11-23T01:14:08.1353314Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1353812Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1354376Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1354926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2 2022-11-23T01:14:08.1355776Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1356256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 6 2022-11-23T01:14:08.1356625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3 2022-11-23T01:14:08.1357127Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1357524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 7 2022-11-23T01:14:08.1357994Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1358560Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1358951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 3 2022-11-23T01:14:08.1359314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 7 2022-11-23T01:14:08.1359656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 6 2022-11-23T01:14:08.1360139Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1360787Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1361178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1 2022-11-23T01:14:08.1361654Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1362154Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1362540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 5 2022-11-23T01:14:08.1362888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 4 2022-11-23T01:14:08.1363370Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1363762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 2 2022-11-23T01:14:08.1364124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0 2022-11-23T01:14:08.1364599Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1365111Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1365620Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1366122Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1366495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 7 2022-11-23T01:14:08.1366856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0 2022-11-23T01:14:08.1367218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 6 2022-11-23T01:14:08.1367701Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1368198Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1368708Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1369099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 5 2022-11-23T01:14:08.1369464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 4 2022-11-23T01:14:08.1369930Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1370324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 2 2022-11-23T01:14:08.1370687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 3 2022-11-23T01:14:08.1371094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1 2022-11-23T01:14:08.1371588Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1372098Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1372614Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1373108Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1373661Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1374053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0 2022-11-23T01:14:08.1374422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 2 2022-11-23T01:14:08.1374772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1 2022-11-23T01:14:08.1375256Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1375766Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1376159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 5 2022-11-23T01:14:08.1376624Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1377011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 7 2022-11-23T01:14:08.1377373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 3 2022-11-23T01:14:08.1377738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 4 2022-11-23T01:14:08.1378084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 6 2022-11-23T01:14:08.1378564Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1379076Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1379455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 4 2022-11-23T01:14:08.1379821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 6 2022-11-23T01:14:08.1380310Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1380826Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1381323Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1381841Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1382350Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1382741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0 2022-11-23T01:14:08.1383097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 2 2022-11-23T01:14:08.1383466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 3 2022-11-23T01:14:08.1383856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1 2022-11-23T01:14:08.1384345Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1384722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 7 2022-11-23T01:14:08.1385080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 5 2022-11-23T01:14:08.1385570Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1386082Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1386619Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1387131Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1387642Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1388035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 5 2022-11-23T01:14:08.1388511Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1388909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 2 2022-11-23T01:14:08.1389282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 3 2022-11-23T01:14:08.1389758Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1390270Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1390665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 4 2022-11-23T01:14:08.1391038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 6 2022-11-23T01:14:08.1391394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 7 2022-11-23T01:14:08.1391758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1 2022-11-23T01:14:08.1392198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0 2022-11-23T01:14:08.1392698Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1393204Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1393599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0 2022-11-23T01:14:08.1394091Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1394604Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1394987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 4 2022-11-23T01:14:08.1395352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 6 2022-11-23T01:14:08.1395845Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1396355Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1396774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 7 2022-11-23T01:14:08.1397265Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1397780Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1398159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 5 2022-11-23T01:14:08.1398523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1 2022-11-23T01:14:08.1398917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 2 2022-11-23T01:14:08.1399409Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1399796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 3 2022-11-23T01:14:08.1400284Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1400986Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1401386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 2 2022-11-23T01:14:08.1401861Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1402260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0 2022-11-23T01:14:08.1402631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 3 2022-11-23T01:14:08.1402994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1 2022-11-23T01:14:08.1403476Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1403987Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1404384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 6 2022-11-23T01:14:08.1404872Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1405372Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1405764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 7 2022-11-23T01:14:08.1406131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 5 2022-11-23T01:14:08.1406482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 4 2022-11-23T01:14:08.1406973Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1407486Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1408002Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1408497Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1408897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 2 2022-11-23T01:14:08.1409382Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1409958Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1410345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0 2022-11-23T01:14:08.1410707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 4 2022-11-23T01:14:08.1411069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 3 2022-11-23T01:14:08.1411433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1 2022-11-23T01:14:08.1411946Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1412467Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1412863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 5 2022-11-23T01:14:08.1413232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 7 2022-11-23T01:14:08.1413588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 6 2022-11-23T01:14:08.1414080Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1414601Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1415118Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1415615Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1416127Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1416644Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1417158Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1417655Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1417942Z ok (3.139s) 2022-11-23T01:14:08.1418282Z test_reduce_scatter_1d (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11166 2022-11-23T01:14:08.1418678Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11167 2022-11-23T01:14:08.1419011Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11168 2022-11-23T01:14:08.1419347Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11169 2022-11-23T01:14:08.1419679Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 11170 2022-11-23T01:14:08.1419993Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 11171 2022-11-23T01:14:08.1420323Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 11172 2022-11-23T01:14:08.1420654Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 11173 2022-11-23T01:14:08.1421097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1421439Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1421882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1422233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1422593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T01:14:08.1423060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1423391Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1423805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1424157Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1424488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T01:14:08.1424982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1425298Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1425729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1426076Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1426398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:14:08.1426836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1427169Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1427593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1427926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1428248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T01:14:08.1428709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1429038Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1429449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1429795Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1430117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T01:14:08.1430565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1430897Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1431328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1431672Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1432066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:14:08.1432524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1432857Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1433287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1433616Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1433942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:14:08.1434408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1434722Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1435187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1435534Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1435856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:14:08.1436207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:14:08.1436581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:14:08.1436951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:14:08.1437302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:14:08.1437699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5 2022-11-23T01:14:08.1438066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4 2022-11-23T01:14:08.1438431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7 2022-11-23T01:14:08.1438780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6 2022-11-23T01:14:08.1439270Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1439792Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1440307Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1440919Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1441438Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1441957Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1442471Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1442968Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1443569Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce! 2022-11-23T01:14:08.1444012Z "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!" 2022-11-23T01:14:08.1444566Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce! 2022-11-23T01:14:08.1444978Z "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!" 2022-11-23T01:14:08.1445534Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce! 2022-11-23T01:14:08.1445955Z "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!" 2022-11-23T01:14:08.1446503Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce! 2022-11-23T01:14:08.1446915Z "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!" 2022-11-23T01:14:08.1447464Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce! 2022-11-23T01:14:08.1447937Z "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!" 2022-11-23T01:14:08.1448495Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce! 2022-11-23T01:14:08.1448903Z "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!" 2022-11-23T01:14:08.1449450Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce! 2022-11-23T01:14:08.1449869Z "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!" 2022-11-23T01:14:08.1450460Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce! 2022-11-23T01:14:08.1450872Z "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!" 2022-11-23T01:14:08.1451120Z ok (2.839s) 2022-11-23T01:14:08.1451458Z test_reduce_scatter_nd (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11278 2022-11-23T01:14:08.1451860Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11279 2022-11-23T01:14:08.1452180Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11280 2022-11-23T01:14:08.1452509Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11281 2022-11-23T01:14:08.1452838Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 11282 2022-11-23T01:14:08.1453153Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 11283 2022-11-23T01:14:08.1453481Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 11284 2022-11-23T01:14:08.1453807Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 11285 2022-11-23T01:14:08.1454258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1454580Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1455010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1455359Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1455670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:14:08.1456132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1456469Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1456902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1457237Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1457563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:14:08.1458022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1458334Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1458761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1459102Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1459424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:14:08.1459867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1460196Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1460658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1461005Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1461311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:14:08.1461774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1462103Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1462516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1462892Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1463216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T01:14:08.1463678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1463994Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1464421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1464770Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1465082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T01:14:08.1465536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1465866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1466301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1466632Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1466957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T01:14:08.1467417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1467749Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1468163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1468510Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1468834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T01:14:08.1469187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:14:08.1469559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:14:08.1469935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:14:08.1470306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:14:08.1470658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4 2022-11-23T01:14:08.1471024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5 2022-11-23T01:14:08.1471387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6 2022-11-23T01:14:08.1471939Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1472344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7 2022-11-23T01:14:08.1472841Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1473397Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1473903Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1474414Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1474929Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1475442Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1475973Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1476364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T01:14:08.1476737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-11-23T01:14:08.1477105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 6 2022-11-23T01:14:08.1477454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2022-11-23T01:14:08.1477816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5 2022-11-23T01:14:08.1478186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 7 2022-11-23T01:14:08.1478551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4 2022-11-23T01:14:08.1478902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T01:14:08.1479388Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1479906Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1480413Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1481027Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1481551Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1482057Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1482434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3 2022-11-23T01:14:08.1482796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 7 2022-11-23T01:14:08.1483161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 5 2022-11-23T01:14:08.1483509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T01:14:08.1483988Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1484372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T01:14:08.1484855Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1485230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 4 2022-11-23T01:14:08.1485585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 6 2022-11-23T01:14:08.1485814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2 2022-11-23T01:14:08.1486109Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1486277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2 2022-11-23T01:14:08.1486568Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1486861Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1487021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 4 2022-11-23T01:14:08.1487347Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1487637Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1487808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3 2022-11-23T01:14:08.1488099Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1488388Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1488674Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1488846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 7 2022-11-23T01:14:08.1489016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T01:14:08.1489186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 6 2022-11-23T01:14:08.1489344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T01:14:08.1489511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 5 2022-11-23T01:14:08.1489797Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1490086Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1490378Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1490548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1 2022-11-23T01:14:08.1490720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 5 2022-11-23T01:14:08.1490891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 4 2022-11-23T01:14:08.1491179Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1491457Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1491745Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1491914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2 2022-11-23T01:14:08.1492203Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1492490Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1492695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0 2022-11-23T01:14:08.1492865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 6 2022-11-23T01:14:08.1493036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3 2022-11-23T01:14:08.1493206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 7 2022-11-23T01:14:08.1493481Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1493769Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1494083Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1494373Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1494662Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1494831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 3 2022-11-23T01:14:08.1495000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 7 2022-11-23T01:14:08.1495169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 6 2022-11-23T01:14:08.1495457Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1495745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1496021Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1496187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 5 2022-11-23T01:14:08.1496357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1 2022-11-23T01:14:08.1496526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 4 2022-11-23T01:14:08.1496697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 2 2022-11-23T01:14:08.1496862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0 2022-11-23T01:14:08.1497153Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1497444Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1497614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0 2022-11-23T01:14:08.1497892Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1498180Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1498464Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1498631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 5 2022-11-23T01:14:08.1498804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 4 2022-11-23T01:14:08.1499093Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1499423Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1499597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 7 2022-11-23T01:14:08.1499884Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1500051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 6 2022-11-23T01:14:08.1500209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 2 2022-11-23T01:14:08.1500377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1 2022-11-23T01:14:08.1500695Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1500867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 3 2022-11-23T01:14:08.1501152Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1501318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1 2022-11-23T01:14:08.1501485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 3 2022-11-23T01:14:08.1501773Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1501939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 7 2022-11-23T01:14:08.1502213Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1502501Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1502786Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1502951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0 2022-11-23T01:14:08.1503121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 2 2022-11-23T01:14:08.1503412Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1503702Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1503873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 5 2022-11-23T01:14:08.1504040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 4 2022-11-23T01:14:08.1504316Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1504484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 6 2022-11-23T01:14:08.1504772Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1505059Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1505233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 4 2022-11-23T01:14:08.1505405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 3 2022-11-23T01:14:08.1505569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 6 2022-11-23T01:14:08.1505887Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1506176Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1506460Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1506615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0 2022-11-23T01:14:08.1506785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 2 2022-11-23T01:14:08.1507076Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1507395Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1507567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1 2022-11-23T01:14:08.1507734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 7 2022-11-23T01:14:08.1507904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 5 2022-11-23T01:14:08.1508193Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1508481Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1508752Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1509042Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1509215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 5 2022-11-23T01:14:08.1509386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 2 2022-11-23T01:14:08.1509678Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1509962Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1510247Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1510534Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1510710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 4 2022-11-23T01:14:08.1510882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 3 2022-11-23T01:14:08.1511040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 6 2022-11-23T01:14:08.1511208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 7 2022-11-23T01:14:08.1511372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1 2022-11-23T01:14:08.1511535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0 2022-11-23T01:14:08.1511913Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1512218Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1512389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0 2022-11-23T01:14:08.1512714Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1513007Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1513169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 4 2022-11-23T01:14:08.1513465Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1513633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 6 2022-11-23T01:14:08.1513928Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1514247Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1514423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 5 2022-11-23T01:14:08.1514714Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1514886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 7 2022-11-23T01:14:08.1515053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1 2022-11-23T01:14:08.1515207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 3 2022-11-23T01:14:08.1515371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 2 2022-11-23T01:14:08.1515665Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1515953Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1516250Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1516417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 6 2022-11-23T01:14:08.1516582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 2 2022-11-23T01:14:08.1516874Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1517167Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1517339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 7 2022-11-23T01:14:08.1517619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1517913Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1518085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0 2022-11-23T01:14:08.1518253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1 2022-11-23T01:14:08.1518545Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1518713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 3 2022-11-23T01:14:08.1518881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 4 2022-11-23T01:14:08.1519044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 5 2022-11-23T01:14:08.1519366Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1519645Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1519935Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1520223Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1520509Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1520816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 4 2022-11-23T01:14:08.1520988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 2 2022-11-23T01:14:08.1521157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 3 2022-11-23T01:14:08.1521451Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1521741Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1522017Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1522187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0 2022-11-23T01:14:08.1522356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1 2022-11-23T01:14:08.1522520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 5 2022-11-23T01:14:08.1522687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 7 2022-11-23T01:14:08.1522860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 6 2022-11-23T01:14:08.1523149Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1523442Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1523728Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1524016Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1524297Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1524589Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1524880Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1525171Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1525552Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce! 2022-11-23T01:14:08.1525711Z "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!" 2022-11-23T01:14:08.1526087Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce! 2022-11-23T01:14:08.1526243Z "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!" 2022-11-23T01:14:08.1526660Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce! 2022-11-23T01:14:08.1526816Z "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!" 2022-11-23T01:14:08.1527172Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce! 2022-11-23T01:14:08.1527326Z "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!" 2022-11-23T01:14:08.1527695Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce! 2022-11-23T01:14:08.1527882Z "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!" 2022-11-23T01:14:08.1528249Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce! 2022-11-23T01:14:08.1528402Z "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!" 2022-11-23T01:14:08.1528767Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce! 2022-11-23T01:14:08.1528920Z "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!" 2022-11-23T01:14:08.1529284Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce! 2022-11-23T01:14:08.1529438Z "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!" 2022-11-23T01:14:08.1529494Z ok (3.138s) 2022-11-23T01:14:08.1529733Z test_reduce_scatter_uneven (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11462 2022-11-23T01:14:08.1529890Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11463 2022-11-23T01:14:08.1530044Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11464 2022-11-23T01:14:08.1530197Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11465 2022-11-23T01:14:08.1530344Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 11466 2022-11-23T01:14:08.1530492Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 11467 2022-11-23T01:14:08.1530640Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 11468 2022-11-23T01:14:08.1530784Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 11469 2022-11-23T01:14:08.1531059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1531185Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1531466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1531603Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1531766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T01:14:08.1532035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1532159Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1532436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1532556Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1532718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T01:14:08.1533020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1533144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1533421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1533552Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1533714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:14:08.1533979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1534118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1534392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1534526Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1534686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:14:08.1534950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1535070Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1535342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1535474Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1535635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:14:08.1535888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1536008Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1536279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1536410Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1536567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:14:08.1536830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1536954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1537225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1537345Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1537506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T01:14:08.1537772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1537891Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1538161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1538291Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1538447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T01:14:08.1538624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:14:08.1538800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:14:08.1538963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:14:08.1539164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:14:08.1539336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4 2022-11-23T01:14:08.1539507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6 2022-11-23T01:14:08.1539673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7 2022-11-23T01:14:08.1539972Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1540142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5 2022-11-23T01:14:08.1540465Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1540761Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1541039Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1541327Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1541614Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1541908Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1542201Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1542577Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce! 2022-11-23T01:14:08.1542736Z "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!" 2022-11-23T01:14:08.1543107Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce! 2022-11-23T01:14:08.1543262Z "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!" 2022-11-23T01:14:08.1543629Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce! 2022-11-23T01:14:08.1543770Z "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!" 2022-11-23T01:14:08.1544137Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce! 2022-11-23T01:14:08.1544294Z "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!" 2022-11-23T01:14:08.1544656Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce! 2022-11-23T01:14:08.1544808Z "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!" 2022-11-23T01:14:08.1545174Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce! 2022-11-23T01:14:08.1545327Z "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!" 2022-11-23T01:14:08.1545693Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce! 2022-11-23T01:14:08.1545845Z "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!" 2022-11-23T01:14:08.1546249Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce! 2022-11-23T01:14:08.1546394Z "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!" 2022-11-23T01:14:08.1546461Z ok (2.839s) 2022-11-23T01:14:08.1546684Z test_scatter_1d (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11574 2022-11-23T01:14:08.1546842Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11575 2022-11-23T01:14:08.1546996Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11576 2022-11-23T01:14:08.1547183Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11577 2022-11-23T01:14:08.1547331Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 11578 2022-11-23T01:14:08.1547486Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 11579 2022-11-23T01:14:08.1547636Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 11580 2022-11-23T01:14:08.1547772Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 11581 2022-11-23T01:14:08.1548048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1548169Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1548448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1548581Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1548747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T01:14:08.1549019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1549142Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1549407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1549542Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1549704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:14:08.1549974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1550095Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1550373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1550505Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1550666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T01:14:08.1550938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1551045Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1551319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1551453Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1551613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:14:08.1551959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1552086Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1552362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1552528Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1552677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T01:14:08.1552944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1553064Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1553342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1553472Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1553667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T01:14:08.1553933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1554053Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1554325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1554444Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1554603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:14:08.1554867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1554987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1555257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1555393Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1555551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:14:08.1555729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:14:08.1555892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:14:08.1556069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:14:08.1556241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5 2022-11-23T01:14:08.1556411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:14:08.1556582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4 2022-11-23T01:14:08.1556752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6 2022-11-23T01:14:08.1556919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7 2022-11-23T01:14:08.1557216Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1557509Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1557794Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1558087Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1558376Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1558665Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1558954Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1559278Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1559348Z ok (2.738s) 2022-11-23T01:14:08.1559573Z test_scatter_nd (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11686 2022-11-23T01:14:08.1559729Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11687 2022-11-23T01:14:08.1559870Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11688 2022-11-23T01:14:08.1560020Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11689 2022-11-23T01:14:08.1560200Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 11690 2022-11-23T01:14:08.1560349Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 11691 2022-11-23T01:14:08.1560498Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 11692 2022-11-23T01:14:08.1560786Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 11693 2022-11-23T01:14:08.1561064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1561186Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1561467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1561589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1561753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:14:08.1562027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1562146Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1562424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1562558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1562720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:14:08.1562987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1563093Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1563366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1563499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1563660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T01:14:08.1563924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1564043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1564316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1564446Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1564605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T01:14:08.1564859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1564979Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1565252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1565386Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1565594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T01:14:08.1565860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1565978Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1566249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1566367Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1566526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T01:14:08.1566828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1566950Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1567222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1567351Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1567510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:14:08.1567771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1567889Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1568152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1568284Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1568442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:14:08.1568619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:14:08.1568795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:14:08.1568967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:14:08.1569140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:14:08.1569310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4 2022-11-23T01:14:08.1569466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5 2022-11-23T01:14:08.1569634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7 2022-11-23T01:14:08.1569804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6 2022-11-23T01:14:08.1570105Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1570403Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1570694Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1570983Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1571274Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1571564Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1571858Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1572047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2022-11-23T01:14:08.1572335Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1572502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 6 2022-11-23T01:14:08.1572673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T01:14:08.1572844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-11-23T01:14:08.1573016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5 2022-11-23T01:14:08.1573212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 7 2022-11-23T01:14:08.1573379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4 2022-11-23T01:14:08.1573548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T01:14:08.1573831Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1573998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T01:14:08.1574289Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1574581Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1574750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T01:14:08.1575044Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1575336Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1575626Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1575918Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1576086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 5 2022-11-23T01:14:08.1576243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3 2022-11-23T01:14:08.1576532Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1576701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 7 2022-11-23T01:14:08.1576872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 4 2022-11-23T01:14:08.1577043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 6 2022-11-23T01:14:08.1577213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2 2022-11-23T01:14:08.1577502Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1577792Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1578079Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1578354Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1578524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2 2022-11-23T01:14:08.1578839Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1579013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3 2022-11-23T01:14:08.1579181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 7 2022-11-23T01:14:08.1579467Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1579754Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1579950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T01:14:08.1580121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 4 2022-11-23T01:14:08.1580398Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1580565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 6 2022-11-23T01:14:08.1580736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 5 2022-11-23T01:14:08.1580910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T01:14:08.1581203Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1581490Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1581782Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1581955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1 2022-11-23T01:14:08.1582244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1582532Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1582689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0 2022-11-23T01:14:08.1582860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 4 2022-11-23T01:14:08.1583147Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1583438Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1583608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 6 2022-11-23T01:14:08.1583896Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1584067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2 2022-11-23T01:14:08.1584236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 5 2022-11-23T01:14:08.1584404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 7 2022-11-23T01:14:08.1584681Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1584851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3 2022-11-23T01:14:08.1585139Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1585334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 3 2022-11-23T01:14:08.1585622Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1585914Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1586086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 6 2022-11-23T01:14:08.1586374Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1586690Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1586846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1 2022-11-23T01:14:08.1587137Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1587306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 5 2022-11-23T01:14:08.1587595Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1587766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 4 2022-11-23T01:14:08.1587936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 7 2022-11-23T01:14:08.1588104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0 2022-11-23T01:14:08.1588274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 2 2022-11-23T01:14:08.1588560Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1588850Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1589126Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1589292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0 2022-11-23T01:14:08.1589460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 2 2022-11-23T01:14:08.1589749Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1589918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 5 2022-11-23T01:14:08.1590203Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1590494Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1590779Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1590947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 4 2022-11-23T01:14:08.1591102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 7 2022-11-23T01:14:08.1591390Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1591561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 6 2022-11-23T01:14:08.1591732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 3 2022-11-23T01:14:08.1592031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1 2022-11-23T01:14:08.1592332Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1592621Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1592908Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1593076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1 2022-11-23T01:14:08.1593263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 5 2022-11-23T01:14:08.1593555Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1593848Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1594018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 7 2022-11-23T01:14:08.1594307Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1594593Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1594881Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1595054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0 2022-11-23T01:14:08.1595224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 2 2022-11-23T01:14:08.1595393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 3 2022-11-23T01:14:08.1595548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 4 2022-11-23T01:14:08.1595838Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1596006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 6 2022-11-23T01:14:08.1596293Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1596460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 4 2022-11-23T01:14:08.1596630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 6 2022-11-23T01:14:08.1596918Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1597086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0 2022-11-23T01:14:08.1597373Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1597649Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1597818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 2 2022-11-23T01:14:08.1598105Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1598392Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1598560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1 2022-11-23T01:14:08.1598757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 3 2022-11-23T01:14:08.1599045Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1599212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 5 2022-11-23T01:14:08.1599378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 7 2022-11-23T01:14:08.1599668Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1599944Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1600260Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1600437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 5 2022-11-23T01:14:08.1600829Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1601003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 7 2022-11-23T01:14:08.1601171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 4 2022-11-23T01:14:08.1601338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 6 2022-11-23T01:14:08.1601629Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1601921Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1602201Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1602486Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1602660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 2 2022-11-23T01:14:08.1602834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 3 2022-11-23T01:14:08.1603004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0 2022-11-23T01:14:08.1603170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1 2022-11-23T01:14:08.1603465Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1603761Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1604057Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1604213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0 2022-11-23T01:14:08.1604378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1 2022-11-23T01:14:08.1604667Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1604958Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1605250Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1605537Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1605759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 5 2022-11-23T01:14:08.1605933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 7 2022-11-23T01:14:08.1606224Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1606395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 4 2022-11-23T01:14:08.1606552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 6 2022-11-23T01:14:08.1606752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 2 2022-11-23T01:14:08.1606920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 3 2022-11-23T01:14:08.1607215Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1607509Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1607679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 2 2022-11-23T01:14:08.1607847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 3 2022-11-23T01:14:08.1608140Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1608428Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1608711Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1608885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 7 2022-11-23T01:14:08.1609177Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1609349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 6 2022-11-23T01:14:08.1609640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1609932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1610101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0 2022-11-23T01:14:08.1610270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1 2022-11-23T01:14:08.1610436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 4 2022-11-23T01:14:08.1610592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 5 2022-11-23T01:14:08.1610882Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1611175Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1611468Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1611639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 4 2022-11-23T01:14:08.1611815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 5 2022-11-23T01:14:08.1612106Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1612429Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1612602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0 2022-11-23T01:14:08.1612891Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1613168Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1613340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 2 2022-11-23T01:14:08.1613534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1 2022-11-23T01:14:08.1613702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 3 2022-11-23T01:14:08.1613998Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1614169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 7 2022-11-23T01:14:08.1614336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 6 2022-11-23T01:14:08.1614630Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1614920Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1615201Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1615491Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1615784Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1616073Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1616360Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1616651Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1616720Z ok (3.139s) 2022-11-23T01:14:08.1616949Z test_scatter_uneven (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11870 2022-11-23T01:14:08.1617106Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11871 2022-11-23T01:14:08.1617248Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11872 2022-11-23T01:14:08.1617403Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11873 2022-11-23T01:14:08.1617555Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 11874 2022-11-23T01:14:08.1617707Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 11875 2022-11-23T01:14:08.1617858Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 11876 2022-11-23T01:14:08.1618008Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 11877 2022-11-23T01:14:08.1618282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1618407Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1618689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1618809Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1619000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:14:08.1619271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1619393Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1619669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1619801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1619963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:14:08.1620270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1620378Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1620654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1620785Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1620951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:14:08.1621214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1621334Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1621605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1621738Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1621897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T01:14:08.1622152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1622272Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1622540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1622673Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1622834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T01:14:08.1623103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1623224Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1623509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1623628Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1623790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:14:08.1624059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1624183Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1624460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1624590Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1624750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T01:14:08.1625018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1625136Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1625430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1625563Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1625720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T01:14:08.1625896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:14:08.1626072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:14:08.1626245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:14:08.1626414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:14:08.1626612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6 2022-11-23T01:14:08.1626769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4 2022-11-23T01:14:08.1626939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5 2022-11-23T01:14:08.1627110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7 2022-11-23T01:14:08.1627409Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1627704Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1627997Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1628288Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1628576Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1628870Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1629159Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1629439Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1629507Z ok (2.838s) 2022-11-23T01:14:08.1629717Z test_device_mesh_2d (__main__.DeviceMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11982 2022-11-23T01:14:08.1629875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11983 2022-11-23T01:14:08.1630030Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11984 2022-11-23T01:14:08.1630182Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11985 2022-11-23T01:14:08.1630335Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 11986 2022-11-23T01:14:08.1630483Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 11987 2022-11-23T01:14:08.1630620Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 11988 2022-11-23T01:14:08.1630767Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 11989 2022-11-23T01:14:08.1631043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1631165Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1631444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1631580Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1631742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:14:08.1632106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1632230Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1632497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1632628Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1632788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T01:14:08.1633058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1633208Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1633482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1633614Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1633774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T01:14:08.1634028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1634147Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1634420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1634551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1634711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:14:08.1634975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1635094Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1635366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1635497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1635643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:14:08.1635908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1636027Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1636298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1636430Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1636587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T01:14:08.1636853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1636972Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1637232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1637362Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1637518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T01:14:08.1637782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1637903Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1638171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1638341Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1638501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:14:08.1638677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:14:08.1638840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:14:08.1639014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:14:08.1639184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:14:08.1639352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5 2022-11-23T01:14:08.1639548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6 2022-11-23T01:14:08.1639714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4 2022-11-23T01:14:08.1639888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7 2022-11-23T01:14:08.1640187Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1640482Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1640873Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1641165Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1641456Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1641749Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1642039Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1642325Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1642497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T01:14:08.1642668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5 2022-11-23T01:14:08.1642842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2022-11-23T01:14:08.1643013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 7 2022-11-23T01:14:08.1643174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 6 2022-11-23T01:14:08.1643343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4 2022-11-23T01:14:08.1643509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-11-23T01:14:08.1643678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T01:14:08.1643968Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1644261Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1644432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T01:14:08.1644721Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1645054Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1645331Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1645622Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1645795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 5 2022-11-23T01:14:08.1646083Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1646409Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1646580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 7 2022-11-23T01:14:08.1646752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 6 2022-11-23T01:14:08.1646920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 4 2022-11-23T01:14:08.1647091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2 2022-11-23T01:14:08.1647247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T01:14:08.1647413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3 2022-11-23T01:14:08.1647703Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1647990Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1648276Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1648575Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1648870Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1649040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 7 2022-11-23T01:14:08.1649209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 6 2022-11-23T01:14:08.1649367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3 2022-11-23T01:14:08.1649535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 4 2022-11-23T01:14:08.1649822Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1650110Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1650280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2 2022-11-23T01:14:08.1650566Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1650735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 5 2022-11-23T01:14:08.1650904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T01:14:08.1651073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T01:14:08.1651364Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1651667Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1651955Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1664794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1665180Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1665366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 6 2022-11-23T01:14:08.1665544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0 2022-11-23T01:14:08.1665843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 4 2022-11-23T01:14:08.1666149Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1666444Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1666620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 5 2022-11-23T01:14:08.1666902Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1667075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 7 2022-11-23T01:14:08.1667248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1 2022-11-23T01:14:08.1667422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2 2022-11-23T01:14:08.1667593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3 2022-11-23T01:14:08.1667892Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1668186Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1668482Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1668775Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1669054Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1669349Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1669635Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1669927Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1669995Z ok (2.956s) 2022-11-23T01:14:08.1670221Z test_device_mesh_2d_from_dim_groups (__main__.DeviceMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12118 2022-11-23T01:14:08.1670379Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12119 2022-11-23T01:14:08.1670532Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12120 2022-11-23T01:14:08.1670685Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12121 2022-11-23T01:14:08.1670824Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 12122 2022-11-23T01:14:08.1670978Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 12123 2022-11-23T01:14:08.1671127Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 12124 2022-11-23T01:14:08.1671313Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 12125 2022-11-23T01:14:08.1671592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1671716Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1672099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1672236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1672388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T01:14:08.1672693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1672816Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1673095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1673229Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1673395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:14:08.1673664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1673786Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1674059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1674181Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1674341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T01:14:08.1674610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1674733Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1675007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1675141Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1675301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:14:08.1675566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1675685Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1675946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1676078Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1676239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T01:14:08.1676504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1676624Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1676895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1677027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1677187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:14:08.1677445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1677565Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1677894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1678026Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1678184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:14:08.1678448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1678567Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1678838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1679000Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1679147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T01:14:08.1679325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:14:08.1679507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:14:08.1679682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:14:08.1679855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:14:08.1680026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5 2022-11-23T01:14:08.1680196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6 2022-11-23T01:14:08.1680368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7 2022-11-23T01:14:08.1680776Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1680947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4 2022-11-23T01:14:08.1681253Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1681551Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1681841Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1682130Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1682420Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1682716Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1683009Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1683180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 7 2022-11-23T01:14:08.1683341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5 2022-11-23T01:14:08.1683513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T01:14:08.1683681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4 2022-11-23T01:14:08.1683850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 6 2022-11-23T01:14:08.1684024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2022-11-23T01:14:08.1684191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T01:14:08.1684415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-11-23T01:14:08.1684715Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1685009Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1685284Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1685574Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1685904Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1686193Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1686366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 5 2022-11-23T01:14:08.1686537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T01:14:08.1686709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 4 2022-11-23T01:14:08.1686876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 6 2022-11-23T01:14:08.1687045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2 2022-11-23T01:14:08.1687323Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1687618Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1687786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 7 2022-11-23T01:14:08.1687957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T01:14:08.1688125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3 2022-11-23T01:14:08.1688417Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1688707Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1688995Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1689283Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1689569Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1689848Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1690015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 5 2022-11-23T01:14:08.1690302Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1690471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 4 2022-11-23T01:14:08.1690640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 6 2022-11-23T01:14:08.1690811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3 2022-11-23T01:14:08.1690978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2 2022-11-23T01:14:08.1691300Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1691471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 7 2022-11-23T01:14:08.1691626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T01:14:08.1691798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T01:14:08.1692093Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1692381Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1692700Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1692868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 5 2022-11-23T01:14:08.1693156Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1693326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0 2022-11-23T01:14:08.1693612Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1693898Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1694170Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1694342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 6 2022-11-23T01:14:08.1694511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 4 2022-11-23T01:14:08.1694802Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1694971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 7 2022-11-23T01:14:08.1695139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1 2022-11-23T01:14:08.1695310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2 2022-11-23T01:14:08.1695477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3 2022-11-23T01:14:08.1695767Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1696044Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1696334Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1696626Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1696916Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1697199Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1697486Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1697776Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1697845Z ok (2.939s) 2022-11-23T01:14:08.1698106Z test_device_mesh_dim_groups_error (__main__.DeviceMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12254 2022-11-23T01:14:08.1698254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12255 2022-11-23T01:14:08.1698408Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12256 2022-11-23T01:14:08.1698560Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12257 2022-11-23T01:14:08.1698712Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 12258 2022-11-23T01:14:08.1698861Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 12259 2022-11-23T01:14:08.1699012Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 12260 2022-11-23T01:14:08.1699191Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 12261 2022-11-23T01:14:08.1699469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1699593Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1699862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1699996Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1700160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T01:14:08.1700433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1700554Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1700834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1700967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1701130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T01:14:08.1701387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1701507Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1701781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1701915Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1702072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:14:08.1702337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1702457Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1702729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1702863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1703009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:14:08.1703273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1703396Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1703669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1703804Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1703964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:14:08.1704227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1704373Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1704633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1704765Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1704924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T01:14:08.1705187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1705305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1705577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1705739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1705896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:14:08.1706161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1706268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1706539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1706668Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1706824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T01:14:08.1706999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:14:08.1707176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:14:08.1707348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:14:08.1707521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:14:08.1707679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4 2022-11-23T01:14:08.1707851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5 2022-11-23T01:14:08.1708020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7 2022-11-23T01:14:08.1708320Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1708489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6 2022-11-23T01:14:08.1708785Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1709078Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1709369Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1709655Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1709943Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1710218Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1710510Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1710684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T01:14:08.1710887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2022-11-23T01:14:08.1711061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5 2022-11-23T01:14:08.1711228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 6 2022-11-23T01:14:08.1711396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4 2022-11-23T01:14:08.1711566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 7 2022-11-23T01:14:08.1711733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T01:14:08.1712114Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1712319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-11-23T01:14:08.1712616Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1712786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T01:14:08.1712957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2 2022-11-23T01:14:08.1713246Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1713536Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1713823Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1714116Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1714412Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1714570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 5 2022-11-23T01:14:08.1714741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 4 2022-11-23T01:14:08.1714909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 6 2022-11-23T01:14:08.1715076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 7 2022-11-23T01:14:08.1715365Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1715536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3 2022-11-23T01:14:08.1715702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T01:14:08.1715997Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1716291Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1716566Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1716853Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1717143Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1717315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 4 2022-11-23T01:14:08.1717604Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1717804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 5 2022-11-23T01:14:08.1717978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 6 2022-11-23T01:14:08.1718146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 7 2022-11-23T01:14:08.1718435Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1718711Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1718910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2 2022-11-23T01:14:08.1719079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3 2022-11-23T01:14:08.1719249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T01:14:08.1719414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T01:14:08.1719708Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1719997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1720166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0 2022-11-23T01:14:08.1720456Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1720831Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1721125Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1721416Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1721587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 4 2022-11-23T01:14:08.1721757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 5 2022-11-23T01:14:08.1721928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 6 2022-11-23T01:14:08.1722095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 7 2022-11-23T01:14:08.1722389Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1722677Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1722847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1 2022-11-23T01:14:08.1723003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2 2022-11-23T01:14:08.1723170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3 2022-11-23T01:14:08.1723457Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1723746Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1724038Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1724326Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1724659Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1724953Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1725241Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1725529Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1725584Z ok (2.943s) 2022-11-23T01:14:08.1725799Z test_device_mesh_nd (__main__.DeviceMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12390 2022-11-23T01:14:08.1725997Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12391 2022-11-23T01:14:08.1726153Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12392 2022-11-23T01:14:08.1726312Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12393 2022-11-23T01:14:08.1726461Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 12394 2022-11-23T01:14:08.1726613Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 12395 2022-11-23T01:14:08.1726765Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 12396 2022-11-23T01:14:08.1726902Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 12397 2022-11-23T01:14:08.1727178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1727303Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1727588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1727723Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1727886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:14:08.1728155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1728278Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1728542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1728677Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1728837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T01:14:08.1729105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1729223Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1729499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1729632Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1729795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:14:08.1730060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1730168Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1730444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1730576Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1730734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T01:14:08.1731026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1731149Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1731421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1731551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1731696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:14:08.1731960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1732080Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1732384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1732514Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1732674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T01:14:08.1732939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1733057Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1733328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1733447Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1733603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:14:08.1733867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:14:08.1733986Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:14:08.1734259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:14:08.1734392Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:14:08.1734548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T01:14:08.1734723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:14:08.1734886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:14:08.1735057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:14:08.1735231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4 2022-11-23T01:14:08.1735401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5 2022-11-23T01:14:08.1735570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:14:08.1735738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7 2022-11-23T01:14:08.1736037Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1736207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6 2022-11-23T01:14:08.1736497Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1736779Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1737066Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1737354Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1737676Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1737970Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1738258Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes. 2022-11-23T01:14:08.1738429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 6 2022-11-23T01:14:08.1738598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-11-23T01:14:08.1738798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T01:14:08.1738965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2022-11-23T01:14:08.1739127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 7 2022-11-23T01:14:08.1739297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5 2022-11-23T01:14:08.1739462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4 2022-11-23T01:14:08.1739628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T01:14:08.1739919Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1740208Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1740504Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1740800Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1740971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3 2022-11-23T01:14:08.1741126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 7 2022-11-23T01:14:08.1741293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T01:14:08.1741581Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1741746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 5 2022-11-23T01:14:08.1742034Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1742323Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1742609Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes. 2022-11-23T01:14:08.1742774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T01:14:08.1742944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 4 2022-11-23T01:14:08.1743101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2 2022-11-23T01:14:08.1743266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 6 2022-11-23T01:14:08.1743561Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1743850Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1744061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2 2022-11-23T01:14:08.1744236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 6 2022-11-23T01:14:08.1744528Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1744819Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1745106Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1745425Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1745698Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1745869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3 2022-11-23T01:14:08.1746039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 4 2022-11-23T01:14:08.1746211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T01:14:08.1746376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 7 2022-11-23T01:14:08.1746666Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes. 2022-11-23T01:14:08.1746834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 5 2022-11-23T01:14:08.1747005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T01:14:08.1747295Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1747572Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1747865Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1748154Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1748323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1 2022-11-23T01:14:08.1748492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2 2022-11-23T01:14:08.1748663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 6 2022-11-23T01:14:08.1748951Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1749242Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1749530Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1749699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 4 2022-11-23T01:14:08.1749976Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes. 2022-11-23T01:14:08.1750143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0 2022-11-23T01:14:08.1750315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 5 2022-11-23T01:14:08.1750482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 7 2022-11-23T01:14:08.1750676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3 2022-11-23T01:14:08.1750968Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1751259Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1751431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 3 2022-11-23T01:14:08.1751720Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1752104Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1752394Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1752686Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1752857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 5 2022-11-23T01:14:08.1753026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1 2022-11-23T01:14:08.1753196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 6 2022-11-23T01:14:08.1753483Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1753772Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes. 2022-11-23T01:14:08.1753943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 4 2022-11-23T01:14:08.1754103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 7 2022-11-23T01:14:08.1754274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0 2022-11-23T01:14:08.1754443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 2 2022-11-23T01:14:08.1754732Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1755023Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1755192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0 2022-11-23T01:14:08.1755362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 2 2022-11-23T01:14:08.1755652Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1755943Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1756111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 7 2022-11-23T01:14:08.1756385Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1756553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 5 2022-11-23T01:14:08.1756843Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1757133Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1757303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 6 2022-11-23T01:14:08.1757628Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes. 2022-11-23T01:14:08.1757798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 4 2022-11-23T01:14:08.1757966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 3 2022-11-23T01:14:08.1758135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1 2022-11-23T01:14:08.1758413Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1758701Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1758907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1 2022-11-23T01:14:08.1759195Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1759364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 5 2022-11-23T01:14:08.1759654Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1759943Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1760229Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1760516Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1760800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0 2022-11-23T01:14:08.1760958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 2 2022-11-23T01:14:08.1761252Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes. 2022-11-23T01:14:08.1761421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 7 2022-11-23T01:14:08.1761591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 3 2022-11-23T01:14:08.1761761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 6 2022-11-23T01:14:08.1761930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 4 2022-11-23T01:14:08.1762222Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1762511Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1762796Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1763070Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1763243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 4 2022-11-23T01:14:08.1763413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0 2022-11-23T01:14:08.1763581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 2 2022-11-23T01:14:08.1763871Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1764161Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1764375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1 2022-11-23T01:14:08.1764667Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1764834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 3 2022-11-23T01:14:08.1765106Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes. 2022-11-23T01:14:08.1765273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 6 2022-11-23T01:14:08.1765444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 7 2022-11-23T01:14:08.1765654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 5 2022-11-23T01:14:08.1765943Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1766232Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1766407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 5 2022-11-23T01:14:08.1766697Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1766984Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1767273Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1767550Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1767725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 4 2022-11-23T01:14:08.1767895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 6 2022-11-23T01:14:08.1768068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 2 2022-11-23T01:14:08.1768356Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1768648Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes. 2022-11-23T01:14:08.1768821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 3 2022-11-23T01:14:08.1768997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 7 2022-11-23T01:14:08.1769166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1 2022-11-23T01:14:08.1769449Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1769622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0 2022-11-23T01:14:08.1769918Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1770089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0 2022-11-23T01:14:08.1770384Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1770679Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1770851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 5 2022-11-23T01:14:08.1771053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 7 2022-11-23T01:14:08.1771345Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1771625Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1771917Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1772089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 4 2022-11-23T01:14:08.1772286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 6 2022-11-23T01:14:08.1772573Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes. 2022-11-23T01:14:08.1772747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1 2022-11-23T01:14:08.1772915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 2 2022-11-23T01:14:08.1773087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 3 2022-11-23T01:14:08.1773378Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1773671Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1773946Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1774118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 7 2022-11-23T01:14:08.1774409Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1774700Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1774871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 6 2022-11-23T01:14:08.1775163Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1775449Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1775740Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes. 2022-11-23T01:14:08.1775911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0 2022-11-23T01:14:08.1776073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1 2022-11-23T01:14:08.1776240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 2 2022-11-23T01:14:08.1776406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 3 2022-11-23T01:14:08.1776572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 4 2022-11-23T01:14:08.1776736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 5 2022-11-23T01:14:08.1777024Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1777314Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1777484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 4 2022-11-23T01:14:08.1777677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 5 2022-11-23T01:14:08.1777958Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1778244Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1778416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0 2022-11-23T01:14:08.1778707Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1778941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1 2022-11-23T01:14:08.1779232Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1779406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 2 2022-11-23T01:14:08.1779574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 3 2022-11-23T01:14:08.1779864Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1780155Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes. 2022-11-23T01:14:08.1780312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 7 2022-11-23T01:14:08.1780478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 6 2022-11-23T01:14:08.1780775Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1781071Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1781365Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1781654Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1781943Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1782232Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1782523Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1782799Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes. 2022-11-23T01:14:08.1782867Z ok (3.139s) 2022-11-23T01:14:08.1782880Z 2022-11-23T01:14:08.1783080Z ---------------------------------------------------------------------- 2022-11-23T01:14:08.1783160Z Ran 19 tests in 56.710s 2022-11-23T01:14:08.1783166Z 2022-11-23T01:14:08.1783228Z OK 2022-11-23T01:14:08.1783233Z 2022-11-23T01:14:08.1783320Z Generating XML reports... 2022-11-23T01:14:08.1783679Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_device_mesh/TEST-DeviceMeshCollectiveTest-20221123011311.xml 2022-11-23T01:14:08.1783996Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_device_mesh/TEST-DeviceMeshTest-20221123011311.xml 2022-11-23T01:14:08.1784003Z 2022-11-23T01:14:08.1784454Z ##[endgroup] 2022-11-23T01:14:08.1784802Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_device_mesh (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_device_mesh_ogec3cgs) 2022-11-23T01:14:08.1784822Z 2022-11-23T01:14:08.1785055Z Running distributed/fsdp/test_fsdp_mixed_precision ... [2022-11-23 01:14:08.079908] 2022-11-23T01:14:08.1785438Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_mixed_precision.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:14:08.080341] 2022-11-23T01:15:33.2970162Z 2022-11-23T01:15:33.2970679Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_mixed_precision 2022-11-23T01:15:33.2974196Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_mixed_precision (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_mixed_precision_02ew3p5d) 2022-11-23T01:15:33.2999003Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision 2022-11-23T01:15:33.2999697Z 2022-11-23T01:15:33.3000136Z Running tests... 2022-11-23T01:15:33.3000770Z ---------------------------------------------------------------------- 2022-11-23T01:15:33.3013386Z test_grads_reduced_precision (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12585 2022-11-23T01:15:33.3014130Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12586 2022-11-23T01:15:33.3014923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3015540Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3016296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3016813Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3017181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3017687Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3018194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3018516Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3019194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3019821Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3020436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3021157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3021554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3021906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3022181Z dist init r=1, world=2 2022-11-23T01:15:33.3022352Z dist init r=0, world=2 2022-11-23T01:15:33.3022568Z skip: Need at least 2 CUDA devices (2.026s) 2022-11-23T01:15:33.3023014Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12613 2022-11-23T01:15:33.3023475Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12614 2022-11-23T01:15:33.3023933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3041888Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3042811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3043459Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3044245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3045139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3045553Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3045986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3046320Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3046664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3047166Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3047739Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3048134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3048486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3048750Z dist init r=0, world=2 2022-11-23T01:15:33.3048920Z dist init r=1, world=2 2022-11-23T01:15:33.3049129Z skip: Need at least 2 CUDA devices (1.915s) 2022-11-23T01:15:33.3049550Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12641 2022-11-23T01:15:33.3049990Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12642 2022-11-23T01:15:33.3052770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3053292Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3054005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3054533Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3054877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3055346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3055679Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3056111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3056452Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3056792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3057293Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3057816Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3058189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3058542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3058805Z dist init r=1, world=2 2022-11-23T01:15:33.3058977Z dist init r=0, world=2 2022-11-23T01:15:33.3059192Z skip: Need at least 2 CUDA devices (1.914s) 2022-11-23T01:15:33.3059638Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12669 2022-11-23T01:15:33.3060116Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12670 2022-11-23T01:15:33.3060656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3060995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3061426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3061782Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3062105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3062587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3062970Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3063388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3063733Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3064074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3064575Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3065081Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3065468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3067730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3068136Z dist init r=1, world=2 2022-11-23T01:15:33.3068420Z dist init r=0, world=2 2022-11-23T01:15:33.3068768Z skip: Need at least 2 CUDA devices (1.915s) 2022-11-23T01:15:33.3069540Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12697 2022-11-23T01:15:33.3070378Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12698 2022-11-23T01:15:33.3070996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3071334Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3071771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3072189Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3072531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3073006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3073341Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3073759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3074109Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3074448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3074947Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3075455Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3075849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3076212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3076475Z dist init r=1, world=2 2022-11-23T01:15:33.3076648Z dist init r=0, world=2 2022-11-23T01:15:33.3076935Z skip: Need at least 2 CUDA devices (1.914s) 2022-11-23T01:15:33.3077380Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12725 2022-11-23T01:15:33.3077838Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12726 2022-11-23T01:15:33.3078295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3078629Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3079094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3079429Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3081596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3082381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3082914Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3083783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3084336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3084881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3085678Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3086550Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3087223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3087776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3088028Z dist init r=0, world=2 2022-11-23T01:15:33.3088214Z dist init r=1, world=2 2022-11-23T01:15:33.3088428Z skip: Need at least 2 CUDA devices (1.914s) 2022-11-23T01:15:33.3088838Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12753 2022-11-23T01:15:33.3089291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12754 2022-11-23T01:15:33.3089753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3090087Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3090509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3090857Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3091201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3091672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3091986Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3092417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3092768Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3093093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3093588Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3094205Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3094597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3094930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3095187Z dist init r=1, world=2 2022-11-23T01:15:33.3095372Z dist init r=0, world=2 2022-11-23T01:15:33.3095569Z skip: Need at least 2 CUDA devices (1.914s) 2022-11-23T01:15:33.3096009Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12781 2022-11-23T01:15:33.3096527Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12782 2022-11-23T01:15:33.3096989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3097310Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3097740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3098091Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3098432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3098885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3099219Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3099648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3099979Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3100315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3100816Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3101340Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3101713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3102061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3102322Z dist init r=0, world=2 2022-11-23T01:15:33.3102510Z dist init r=1, world=2 2022-11-23T01:15:33.3102708Z skip: Need at least 2 CUDA devices (2.015s) 2022-11-23T01:15:33.3103127Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12809 2022-11-23T01:15:33.3103575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12810 2022-11-23T01:15:33.3104016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3104353Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3104784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3105130Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3105454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3105930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3106262Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3106716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3107063Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3107401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3107899Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3108409Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3108798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3109185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3109452Z dist init r=0, world=2 2022-11-23T01:15:33.3109621Z dist init r=1, world=2 2022-11-23T01:15:33.3109836Z skip: Need at least 2 CUDA devices (1.914s) 2022-11-23T01:15:33.3110280Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12837 2022-11-23T01:15:33.3110742Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12838 2022-11-23T01:15:33.3111199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3111534Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3112041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3112380Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3112720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3113196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3113515Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3113940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3114285Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3114622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3115103Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3115625Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3116014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3116366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3116616Z dist init r=0, world=2 2022-11-23T01:15:33.3116800Z dist init r=1, world=2 2022-11-23T01:15:33.3117010Z skip: Need at least 2 CUDA devices (1.914s) 2022-11-23T01:15:33.3117415Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12865 2022-11-23T01:15:33.3117866Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12866 2022-11-23T01:15:33.3118322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3118656Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3119074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3119478Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3119821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3120279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3120838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3121279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3121628Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3122007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3122503Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3123031Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3123423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3123760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3124017Z dist init r=1, world=2 2022-11-23T01:15:33.3124204Z dist init r=0, world=2 2022-11-23T01:15:33.3124403Z skip: Need at least 2 CUDA devices (2.014s) 2022-11-23T01:15:33.3124848Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12893 2022-11-23T01:15:33.3125329Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12894 2022-11-23T01:15:33.3125789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3126109Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3126535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3126884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3127213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3127686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3128018Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3128453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3128784Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3129122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3129618Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3130141Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3130514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3130863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3131130Z dist init r=1, world=2 2022-11-23T01:15:33.3131304Z dist init r=0, world=2 2022-11-23T01:15:33.3131514Z skip: Need at least 2 CUDA devices (2.014s) 2022-11-23T01:15:33.3131940Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12921 2022-11-23T01:15:33.3132433Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12922 2022-11-23T01:15:33.3132876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3133207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3133637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3133973Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3134315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3134817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3135147Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3135564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3135912Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3136250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3136745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3137251Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3137641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3137994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3138243Z dist init r=0, world=2 2022-11-23T01:15:33.3138426Z dist init r=1, world=2 2022-11-23T01:15:33.3138638Z skip: Need at least 2 CUDA devices (1.914s) 2022-11-23T01:15:33.3139080Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12949 2022-11-23T01:15:33.3139539Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12950 2022-11-23T01:15:33.3139996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3140329Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3140747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3141099Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3141440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3141915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3142231Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3142658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3143005Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3143344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3143824Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3144344Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3144733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3145116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3145382Z dist init r=1, world=2 2022-11-23T01:15:33.3145567Z dist init r=0, world=2 2022-11-23T01:15:33.3145780Z skip: Need at least 2 CUDA devices (1.914s) 2022-11-23T01:15:33.3146187Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12977 2022-11-23T01:15:33.3146640Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12978 2022-11-23T01:15:33.3147102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3147461Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3147888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3148243Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3148585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3149045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3149374Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3149808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3150158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3150483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3150982Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3151507Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3151955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3152313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3152571Z dist init r=1, world=2 2022-11-23T01:15:33.3152755Z dist init r=0, world=2 2022-11-23T01:15:33.3152951Z skip: Need at least 2 CUDA devices (1.914s) 2022-11-23T01:15:33.3153398Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13005 2022-11-23T01:15:33.3153875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13006 2022-11-23T01:15:33.3154320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3154654Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3155086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3155433Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3155759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3156229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3156564Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3156997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3157327Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3157708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3158207Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3158715Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3159105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3159459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3159722Z dist init r=1, world=2 2022-11-23T01:15:33.3159924Z dist init r=0, world=2 2022-11-23T01:15:33.3160136Z skip: Need at least 2 CUDA devices (1.914s) 2022-11-23T01:15:33.3160556Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13033 2022-11-23T01:15:33.3161345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13034 2022-11-23T01:15:33.3161999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3162332Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3162761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3163095Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3163436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3163910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3164242Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3164662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3165019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3165355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3165834Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3166357Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3166748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3167105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3167353Z dist init r=0, world=2 2022-11-23T01:15:33.3167535Z dist init r=1, world=2 2022-11-23T01:15:33.3167742Z skip: Need at least 2 CUDA devices (1.914s) 2022-11-23T01:15:33.3168190Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13061 2022-11-23T01:15:33.3168734Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13062 2022-11-23T01:15:33.3169189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3169517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3169933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3170279Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3170612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3171154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3171475Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3171898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3172243Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3172575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3173059Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3173621Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3174005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3174340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3174604Z dist init r=1, world=2 2022-11-23T01:15:33.3174784Z dist init r=0, world=2 2022-11-23T01:15:33.3174989Z skip: Need at least 2 CUDA devices (1.915s) 2022-11-23T01:15:33.3175405Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13089 2022-11-23T01:15:33.3175872Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13090 2022-11-23T01:15:33.3176545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3176878Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3177293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3177642Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3177976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3178431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3178757Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3179185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3179527Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3179851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3180346Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3180861Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3181249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3181584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3181838Z dist init r=1, world=2 2022-11-23T01:15:33.3182019Z dist init r=0, world=2 2022-11-23T01:15:33.3182211Z skip: Need at least 2 CUDA devices (1.915s) 2022-11-23T01:15:33.3182663Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13117 2022-11-23T01:15:33.3183146Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13118 2022-11-23T01:15:33.3183638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3183955Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3184380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3184725Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3185047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3185512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3185840Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3186295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3186625Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3186961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3187455Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3187972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3188344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3188695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3188953Z dist init r=1, world=2 2022-11-23T01:15:33.3189124Z dist init r=0, world=2 2022-11-23T01:15:33.3189329Z skip: Need at least 2 CUDA devices (2.014s) 2022-11-23T01:15:33.3189759Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13145 2022-11-23T01:15:33.3190220Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13146 2022-11-23T01:15:33.3190662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3190990Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3191423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3191758Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3192179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3192652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3192978Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3193390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3193735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3194065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3194549Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3195069Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3195452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3195801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3196047Z dist init r=0, world=2 2022-11-23T01:15:33.3196226Z dist init r=1, world=2 2022-11-23T01:15:33.3196479Z skip: Need at least 2 CUDA devices (1.914s) 2022-11-23T01:15:33.3196919Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13173 2022-11-23T01:15:33.3197407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13174 2022-11-23T01:15:33.3197857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3198186Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3198598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3198974Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3199310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3199781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3200091Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3200514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3201004Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3201327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3201824Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3202348Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3202738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3203073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3203332Z dist init r=0, world=2 2022-11-23T01:15:33.3203514Z dist init r=1, world=2 2022-11-23T01:15:33.3203708Z skip: Need at least 2 CUDA devices (1.914s) 2022-11-23T01:15:33.3204138Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13201 2022-11-23T01:15:33.3204599Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13202 2022-11-23T01:15:33.3205057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3205377Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3205804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3206148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3206481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3206936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3207265Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3207688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3208015Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3208347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3208841Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3209442Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3209822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3210173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3210430Z dist init r=0, world=2 2022-11-23T01:15:33.3210610Z dist init r=1, world=2 2022-11-23T01:15:33.3210806Z skip: Need at least 2 CUDA devices (2.015s) 2022-11-23T01:15:33.3211261Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13229 2022-11-23T01:15:33.3211788Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13230 2022-11-23T01:15:33.3212236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3212562Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3212990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3213336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3213657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3214121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3214453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3214868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3215213Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3215548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3216043Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3216555Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3216942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3217286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3217546Z dist init r=0, world=2 2022-11-23T01:15:33.3217713Z dist init r=1, world=2 2022-11-23T01:15:33.3217918Z skip: Need at least 2 CUDA devices (1.914s) 2022-11-23T01:15:33.3218347Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13257 2022-11-23T01:15:33.3218801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13258 2022-11-23T01:15:33.3219258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3219588Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3220016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3220350Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3220691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3221157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3221471Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3221933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3222278Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3222611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3223091Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3223609Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3224030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3224378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3224622Z dist init r=0, world=2 2022-11-23T01:15:33.3224803Z dist init r=1, world=2 2022-11-23T01:15:33.3225010Z skip: Need at least 2 CUDA devices (1.914s) 2022-11-23T01:15:33.3225446Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13285 2022-11-23T01:15:33.3225925Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13286 2022-11-23T01:15:33.3226381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3226708Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3227122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3227469Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3227805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3228271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3228586Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3229016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3229359Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3229677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3230173Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3230697Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3231081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3231414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3231670Z dist init r=1, world=2 2022-11-23T01:15:33.3231850Z dist init r=0, world=2 2022-11-23T01:15:33.3232123Z skip: Need at least 2 CUDA devices (1.914s) 2022-11-23T01:15:33.3232546Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13313 2022-11-23T01:15:33.3232999Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13314 2022-11-23T01:15:33.3233453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3233772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3234235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3234583Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3234906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3235376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3235709Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3236136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3236497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3236830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3237323Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3237842Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3238213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3238564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3238824Z dist init r=1, world=2 2022-11-23T01:15:33.3238991Z dist init r=0, world=2 2022-11-23T01:15:33.3239199Z skip: Need at least 2 CUDA devices (2.014s) 2022-11-23T01:15:33.3239645Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13341 2022-11-23T01:15:33.3240122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13342 2022-11-23T01:15:33.3240568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3241033Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3241464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3241814Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3242135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3242601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3242933Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3243346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3243690Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3244038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3244532Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3245042Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3245429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3245777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3246028Z dist init r=0, world=2 2022-11-23T01:15:33.3246212Z dist init r=1, world=2 2022-11-23T01:15:33.3246418Z skip: Need at least 2 CUDA devices (2.014s) 2022-11-23T01:15:33.3246910Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13369 2022-11-23T01:15:33.3247353Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13370 2022-11-23T01:15:33.3247813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3248143Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3248560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3248903Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3249281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3249750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3250064Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3250491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3250832Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3251166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3251640Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3252157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3252544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3252877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3253135Z dist init r=1, world=2 2022-11-23T01:15:33.3253312Z dist init r=0, world=2 2022-11-23T01:15:33.3253518Z skip: Need at least 2 CUDA devices (1.914s) 2022-11-23T01:15:33.3253949Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13397 2022-11-23T01:15:33.3254428Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13398 2022-11-23T01:15:33.3254877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3255203Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3255621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3255964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3256305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3256762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3257093Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3257515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3257859Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3258180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3258670Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3259191Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3259596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3259946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3260200Z dist init r=1, world=2 2022-11-23T01:15:33.3260380Z dist init r=0, world=2 2022-11-23T01:15:33.3260575Z skip: Need at least 2 CUDA devices (1.915s) 2022-11-23T01:15:33.3260992Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13425 2022-11-23T01:15:33.3261445Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13426 2022-11-23T01:15:33.3261929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3262256Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3262683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3263028Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3263349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3263821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3264147Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3264567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3264903Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3265237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3265734Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3266242Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3266627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3266973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3267233Z dist init r=1, world=2 2022-11-23T01:15:33.3267401Z dist init r=0, world=2 2022-11-23T01:15:33.3267607Z skip: Need at least 2 CUDA devices (2.015s) 2022-11-23T01:15:33.3268048Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13453 2022-11-23T01:15:33.3268526Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13454 2022-11-23T01:15:33.3268968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3269299Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3269728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3270065Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3270399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3270867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3271197Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3271609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3272035Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3272413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3272902Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3273419Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3273806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3274154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3274431Z dist init r=1, world=2 2022-11-23T01:15:33.3274610Z dist init r=0, world=2 2022-11-23T01:15:33.3274819Z skip: Need at least 2 CUDA devices (1.915s) 2022-11-23T01:15:33.3275231Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13481 2022-11-23T01:15:33.3275693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13482 2022-11-23T01:15:33.3276150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3276479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3276890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3277236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3277578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3278044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3278358Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3278784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3279122Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3279443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3279931Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3280448Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3280979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3281316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3281570Z dist init r=1, world=2 2022-11-23T01:15:33.3281751Z dist init r=0, world=2 2022-11-23T01:15:33.3281947Z skip: Need at least 2 CUDA devices (1.914s) 2022-11-23T01:15:33.3282346Z test_mixed_precision_no_reshard_after_forward (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13509 2022-11-23T01:15:33.3282778Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13510 2022-11-23T01:15:33.3283234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3283552Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3283977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3284323Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3284659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3285175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3285502Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3285931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3286261Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3286592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3287086Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3287656Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3288032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3288381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3288642Z dist init r=1, world=2 2022-11-23T01:15:33.3288807Z dist init r=0, world=2 2022-11-23T01:15:33.3289011Z skip: Need at least 2 CUDA devices (1.914s) 2022-11-23T01:15:33.3289292Z test_mixed_precision_resnet (__main__.TestFSDPMixedPrecisionSharded) 2022-11-23T01:15:33.3289630Z End to end test to ensure mixed precision + auto_wrap works ... skip: no torchvision (0.002s) 2022-11-23T01:15:33.3290057Z test_mp_batchnorm_convert_sync_bn_False (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13537 2022-11-23T01:15:33.3290481Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13538 2022-11-23T01:15:33.3290938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3291257Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3291682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3292033Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3292367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3292819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3293144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3293569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3293912Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3294230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3294723Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3295240Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3295614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3295957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3296209Z dist init r=0, world=2 2022-11-23T01:15:33.3296388Z dist init r=1, world=2 2022-11-23T01:15:33.3296587Z skip: Need at least 2 CUDA devices (2.015s) 2022-11-23T01:15:33.3296972Z test_mp_batchnorm_convert_sync_bn_True (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13565 2022-11-23T01:15:33.3297451Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13566 2022-11-23T01:15:33.3297903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3298231Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3298659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3299002Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3299325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3299788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3300151Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3300577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3300909Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3301245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3301740Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3302246Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3302631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3302977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3303235Z dist init r=0, world=2 2022-11-23T01:15:33.3303401Z dist init r=1, world=2 2022-11-23T01:15:33.3303608Z skip: Need at least 2 CUDA devices (1.916s) 2022-11-23T01:15:33.3303988Z test_mp_embedding_default (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13593 2022-11-23T01:15:33.3304389Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13594 2022-11-23T01:15:33.3304844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3305173Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3305601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3305933Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3306268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3306733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3307051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3307477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3307820Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3308161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3308643Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3309165Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3309554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3309902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3310148Z dist init r=0, world=2 2022-11-23T01:15:33.3310360Z dist init r=1, world=2 2022-11-23T01:15:33.3310569Z skip: Need at least 2 CUDA devices (2.014s) 2022-11-23T01:15:33.3310946Z test_mp_embedding_only_params_and_bufs (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13621 2022-11-23T01:15:33.3311368Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13622 2022-11-23T01:15:33.3311820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3312240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3312694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3313038Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3313375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3313827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3314153Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3314576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3314914Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3315232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3315726Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3316247Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3316631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3316964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3317215Z dist init r=1, world=2 2022-11-23T01:15:33.3317392Z dist init r=0, world=2 2022-11-23T01:15:33.3317584Z skip: Need at least 2 CUDA devices (1.914s) 2022-11-23T01:15:33.3317973Z test_mp_embedding_params_and_reduce_diff (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13649 2022-11-23T01:15:33.3318401Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13650 2022-11-23T01:15:33.3318855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3319173Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3319598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3319947Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3320270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3335497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3335923Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3336419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3336775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3337124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3337614Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3338258Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3338654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3339004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3339256Z dist init r=0, world=2 2022-11-23T01:15:33.3339441Z dist init r=1, world=2 2022-11-23T01:15:33.3339652Z skip: Need at least 2 CUDA devices (2.015s) 2022-11-23T01:15:33.3340020Z test_mp_embedding_reduce (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13677 2022-11-23T01:15:33.3340483Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13678 2022-11-23T01:15:33.3340940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3341270Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3341685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3342031Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3342369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3342827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3343156Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3343588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3343931Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3344254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:15:33.3344752Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3345271Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:15:33.3345657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3345990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:15:33.3346246Z dist init r=0, world=2 2022-11-23T01:15:33.3346425Z dist init r=1, world=2 2022-11-23T01:15:33.3346623Z skip: Need at least 2 CUDA devices (1.914s) 2022-11-23T01:15:33.3347011Z test_grads_reduced_precision (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13705 2022-11-23T01:15:33.3347556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3347888Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3348301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3348649Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3348986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3349469Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T01:15:33.3349859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3350120Z dist init r=0, world=1 2022-11-23T01:15:33.3350329Z skip: Need at least 1 CUDA device (1.910s) 2022-11-23T01:15:33.3350735Z test_mixed_precision_e2e_full_shard (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13719 2022-11-23T01:15:33.3351280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3351611Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3352114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3352465Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3352806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3353350Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T01:15:33.3353723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3353981Z dist init r=0, world=1 2022-11-23T01:15:33.3354189Z skip: Need at least 1 CUDA device (1.810s) 2022-11-23T01:15:33.3354577Z test_mixed_precision_no_reshard_after_forward (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13733 2022-11-23T01:15:33.3355138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:15:33.3355471Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:15:33.3355900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:15:33.3356236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:15:33.3356571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:15:33.3357063Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T01:15:33.3357448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:15:33.3357697Z dist init r=0, world=1 2022-11-23T01:15:33.3357903Z skip: Need at least 1 CUDA device (1.810s) 2022-11-23T01:15:33.3358038Z 2022-11-23T01:15:33.3358238Z ---------------------------------------------------------------------- 2022-11-23T01:15:33.3358468Z Ran 44 tests in 83.323s 2022-11-23T01:15:33.3358580Z 2022-11-23T01:15:33.3358653Z OK (skipped=44) 2022-11-23T01:15:33.3358760Z 2022-11-23T01:15:33.3358844Z Generating XML reports... 2022-11-23T01:15:33.3359348Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionSharded-20221123011409.xml 2022-11-23T01:15:33.3360027Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionUnsharded-20221123011409.xml 2022-11-23T01:15:33.3360332Z 2022-11-23T01:15:33.3360876Z ##[endgroup] 2022-11-23T01:15:33.3361415Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_mixed_precision (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_mixed_precision_02ew3p5d) 2022-11-23T01:15:33.3361693Z 2022-11-23T01:15:33.3361902Z Running distributed/fsdp/test_fsdp_summon_full_params ... [2022-11-23 01:15:33.300786] 2022-11-23T01:15:33.3362448Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_summon_full_params.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:15:33.301116] 2022-11-23T01:16:59.1702426Z 2022-11-23T01:16:59.1703244Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_summon_full_params 2022-11-23T01:16:59.1704263Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_summon_full_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_summon_full_params_nfoc1_sm) 2022-11-23T01:16:59.1705479Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params 2022-11-23T01:16:59.1705830Z 2022-11-23T01:16:59.1705947Z Running tests... 2022-11-23T01:16:59.1706432Z ---------------------------------------------------------------------- 2022-11-23T01:16:59.1761999Z test_cannot_summon_full_params_from_backward (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13758 2022-11-23T01:16:59.1762627Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13759 2022-11-23T01:16:59.1763193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1763799Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1764406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1765038Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1765639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.1766419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1767004Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1767604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1768172Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1768782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.1769701Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1770559Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1771165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.1771806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.1772144Z dist init r=0, world=2 2022-11-23T01:16:59.1772329Z dist init r=1, world=2 2022-11-23T01:16:59.1772643Z skip: Need at least 2 CUDA devices (1.882s) 2022-11-23T01:16:59.1773059Z test_cannot_summon_full_params_from_forward (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13786 2022-11-23T01:16:59.1773462Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13787 2022-11-23T01:16:59.1773953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1774557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1775201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1775549Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1775893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.1776350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1776702Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1777128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1777479Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1777803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.1778655Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1779491Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1780213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.1780848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.1781219Z dist init r=1, world=2 2022-11-23T01:16:59.1781400Z dist init r=0, world=2 2022-11-23T01:16:59.1781596Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:16:59.1781885Z test_named_parameters_buffers_prefix__recurse_False (__main__.TestSummonFullParams) 2022-11-23T01:16:59.1782371Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13814 2022-11-23T01:16:59.1782768Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13815 2022-11-23T01:16:59.1783216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1783548Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1783978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1784325Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1784648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.1785113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1785441Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1785851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1786196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1786532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.1787028Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1787533Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1787919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.1788269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.1788531Z dist init r=1, world=2 2022-11-23T01:16:59.1788700Z dist init r=0, world=2 2022-11-23T01:16:59.1788910Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.1789227Z test_named_parameters_buffers_prefix__recurse_True (__main__.TestSummonFullParams) 2022-11-23T01:16:59.1789614Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13842 2022-11-23T01:16:59.1790003Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13843 2022-11-23T01:16:59.1790455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1790782Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1791195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1791543Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1791876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.1792479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1792793Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1793220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1793564Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1793889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.1794384Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1794973Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1795362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.1795703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.1795966Z dist init r=0, world=2 2022-11-23T01:16:59.1796147Z dist init r=1, world=2 2022-11-23T01:16:59.1796342Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.1796650Z test_named_parameters_buffers_prefix_test_prefix_recurse_False (__main__.TestSummonFullParams) 2022-11-23T01:16:59.1797067Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13870 2022-11-23T01:16:59.1797457Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13871 2022-11-23T01:16:59.1797896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1798227Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1798655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1798987Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1799321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.1799782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1800106Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1800518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1801118Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1801664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.1802363Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1803218Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1803829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.1804467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.1804919Z dist init r=0, world=2 2022-11-23T01:16:59.1805239Z dist init r=1, world=2 2022-11-23T01:16:59.1805606Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:16:59.1806129Z test_named_parameters_buffers_prefix_test_prefix_recurse_True (__main__.TestSummonFullParams) 2022-11-23T01:16:59.1806857Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13898 2022-11-23T01:16:59.1807566Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13899 2022-11-23T01:16:59.1808583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1809183Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1809879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1810446Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1811013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.1811869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1812530Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1813306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1813915Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1814477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.1815371Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1816228Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1816838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.1817411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.1817817Z dist init r=1, world=2 2022-11-23T01:16:59.1818105Z dist init r=0, world=2 2022-11-23T01:16:59.1818445Z skip: Need at least 2 CUDA devices (1.616s) 2022-11-23T01:16:59.1819120Z test_params_are_unflattenned_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13926 2022-11-23T01:16:59.1819866Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13927 2022-11-23T01:16:59.1820634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1821174Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1821868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1822497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1823107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.1823946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1824534Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1825330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1825937Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1826506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.1827407Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1828274Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1828665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.1828997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.1829251Z dist init r=0, world=2 2022-11-23T01:16:59.1829490Z dist init r=1, world=2 2022-11-23T01:16:59.1829691Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.1830112Z test_params_are_unflattenned_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13954 2022-11-23T01:16:59.1830565Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13955 2022-11-23T01:16:59.1831023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1831341Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1831804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1832255Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1832583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.1833052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1833380Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1833804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1834136Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1834473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.1834968Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1835490Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1835864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.1836217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.1836473Z dist init r=1, world=2 2022-11-23T01:16:59.1836638Z dist init r=0, world=2 2022-11-23T01:16:59.1836844Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.1837262Z test_params_are_unflattenned_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13982 2022-11-23T01:16:59.1837712Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13983 2022-11-23T01:16:59.1838153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1838479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1838905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1839240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1839576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.1840043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1840367Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1840930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1841282Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1841616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.1842174Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1842684Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1843070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.1843418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.1843667Z dist init r=1, world=2 2022-11-23T01:16:59.1843850Z dist init r=0, world=2 2022-11-23T01:16:59.1844061Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.1844821Z test_params_are_unflattenned_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14010 2022-11-23T01:16:59.1845304Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14011 2022-11-23T01:16:59.1845759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1846088Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1846521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1846855Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1847194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.1847661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1847977Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1848404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1848751Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1849092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.1849573Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1850092Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1850480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.1850826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.1851076Z dist init r=1, world=2 2022-11-23T01:16:59.1851262Z dist init r=0, world=2 2022-11-23T01:16:59.1851469Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.1851875Z test_params_are_unflattenned_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14038 2022-11-23T01:16:59.1852326Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14039 2022-11-23T01:16:59.1852782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1853113Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1853527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1853876Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1854215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.1854668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1854996Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1855455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1855804Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1856126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.1856621Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1857147Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1857566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.1857900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.1858156Z dist init r=1, world=2 2022-11-23T01:16:59.1858339Z dist init r=0, world=2 2022-11-23T01:16:59.1858539Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.1858956Z test_params_are_unflattenned_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14066 2022-11-23T01:16:59.1859407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14067 2022-11-23T01:16:59.1859863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1860179Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1860609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1860956Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1861281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.1861750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1862080Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1862507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1862843Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1863185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.1863678Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1864188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1864577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.1864927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.1865191Z dist init r=0, world=2 2022-11-23T01:16:59.1865361Z dist init r=1, world=2 2022-11-23T01:16:59.1865572Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.1865991Z test_params_are_unflattenned_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14094 2022-11-23T01:16:59.1866429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14095 2022-11-23T01:16:59.1866892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1867225Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1867688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1868024Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1868367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.1868836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1869169Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1869577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1869925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1870291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.1870776Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1871299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1871687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.1872119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.1872369Z dist init r=0, world=2 2022-11-23T01:16:59.1872549Z dist init r=1, world=2 2022-11-23T01:16:59.1872758Z skip: Need at least 2 CUDA devices (1.618s) 2022-11-23T01:16:59.1873161Z test_params_are_unflattenned_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14122 2022-11-23T01:16:59.1873615Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14123 2022-11-23T01:16:59.1874080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1874415Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1874833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1875179Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1875516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.1875983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1876301Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1876730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1877073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1877396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.1877887Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1878406Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1878792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.1879124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.1879377Z dist init r=1, world=2 2022-11-23T01:16:59.1879562Z dist init r=0, world=2 2022-11-23T01:16:59.1879757Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.1880210Z test_params_count_and_value_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14150 2022-11-23T01:16:59.1880888Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14151 2022-11-23T01:16:59.1881361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1881680Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1882112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1882460Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1882799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.1883319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1883647Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1884077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1884405Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1884745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.1885241Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1885761Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1886140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.1886487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.1886746Z dist init r=1, world=2 2022-11-23T01:16:59.1886911Z dist init r=0, world=2 2022-11-23T01:16:59.1887124Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.1887539Z test_params_count_and_value_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14178 2022-11-23T01:16:59.1887988Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14179 2022-11-23T01:16:59.1888430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1888758Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1889183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1889529Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1889853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.1890324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1890651Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1891061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1891402Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1891736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.1892233Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1892741Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1893125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.1893510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.1893774Z dist init r=1, world=2 2022-11-23T01:16:59.1893941Z dist init r=0, world=2 2022-11-23T01:16:59.1894148Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.1894566Z test_params_count_and_value_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14206 2022-11-23T01:16:59.1895000Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14207 2022-11-23T01:16:59.1895453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1895867Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1896298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1896634Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1896972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.1897438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1897750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1898176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1898523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1898861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.1899346Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1899867Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1900252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.1900600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.1900846Z dist init r=1, world=2 2022-11-23T01:16:59.1901028Z dist init r=0, world=2 2022-11-23T01:16:59.1901238Z skip: Need at least 2 CUDA devices (1.715s) 2022-11-23T01:16:59.1901645Z test_params_count_and_value_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14234 2022-11-23T01:16:59.1902088Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14235 2022-11-23T01:16:59.1902543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1902875Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1903291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1903637Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1903972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.1904424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1904751Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1905178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1905523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1905871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.1906370Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1906889Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1907275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.1907606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.1907864Z dist init r=0, world=2 2022-11-23T01:16:59.1908079Z dist init r=1, world=2 2022-11-23T01:16:59.1908279Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.1908697Z test_params_count_and_value_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14262 2022-11-23T01:16:59.1909147Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14263 2022-11-23T01:16:59.1909603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1909918Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1910347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1910692Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1911012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.1911484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1911817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1912331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1912662Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1912999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.1913502Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1914024Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1914400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.1914752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.1915013Z dist init r=1, world=2 2022-11-23T01:16:59.1915184Z dist init r=0, world=2 2022-11-23T01:16:59.1915397Z skip: Need at least 2 CUDA devices (1.616s) 2022-11-23T01:16:59.1915811Z test_params_count_and_value_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14290 2022-11-23T01:16:59.1916261Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14291 2022-11-23T01:16:59.1916705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1917036Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1917465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1917800Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1918139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.1918648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1918980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1919393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1919743Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1920078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.1920561Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1921289Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1921680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.1922034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.1922283Z dist init r=0, world=2 2022-11-23T01:16:59.1922467Z dist init r=1, world=2 2022-11-23T01:16:59.1922677Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.1923077Z test_params_count_and_value_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14318 2022-11-23T01:16:59.1923522Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14319 2022-11-23T01:16:59.1923975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1924312Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1924726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1925074Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1925411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.1925882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1926192Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1926619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1926962Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1927287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.1927780Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1928302Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1928692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.1929032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.1929289Z dist init r=1, world=2 2022-11-23T01:16:59.1929472Z dist init r=0, world=2 2022-11-23T01:16:59.1929667Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.1930080Z test_params_count_and_value_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14346 2022-11-23T01:16:59.1930530Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14347 2022-11-23T01:16:59.1930987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1931349Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1931783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1932130Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1932468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.1932921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1933249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1933712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1934043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1934381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.1934882Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1935401Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1935776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.1936126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.1936389Z dist init r=1, world=2 2022-11-23T01:16:59.1936559Z dist init r=0, world=2 2022-11-23T01:16:59.1936771Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.1937035Z test_raises_rank0_with_writeback (__main__.TestSummonFullParams) 2022-11-23T01:16:59.1937416Z Tests that ``summon_full_params()`` with both ``rank0_only=True`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14374 2022-11-23T01:16:59.1937792Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14375 2022-11-23T01:16:59.1938243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1938574Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1938988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1939335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1939676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.1940148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1940462Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1940890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1941234Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1941568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.1942049Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1942567Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1942958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.1943296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.1943555Z dist init r=0, world=2 2022-11-23T01:16:59.1943735Z dist init r=1, world=2 2022-11-23T01:16:59.1943976Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:16:59.1944409Z test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14402 2022-11-23T01:16:59.1944962Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14403 2022-11-23T01:16:59.1945419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1945752Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1946168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1946552Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1946890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.1947361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1947672Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1948100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1948445Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1948767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.1949260Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1949787Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1950171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.1950511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.1950768Z dist init r=1, world=2 2022-11-23T01:16:59.1950951Z dist init r=0, world=2 2022-11-23T01:16:59.1951145Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.1951590Z test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14430 2022-11-23T01:16:59.1952133Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14431 2022-11-23T01:16:59.1952597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1952912Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1953340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1953692Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1954031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.1954486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1954815Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1955242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1955585Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1955907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.1956399Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1956954Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1957329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.1957674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.1957932Z dist init r=0, world=2 2022-11-23T01:16:59.1958114Z dist init r=1, world=2 2022-11-23T01:16:59.1958310Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.1958747Z test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14458 2022-11-23T01:16:59.1959254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14459 2022-11-23T01:16:59.1959701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1960032Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1960465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1960955Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1961284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.1961754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1962081Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1962511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1962843Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1963182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.1963681Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1964187Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1964583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.1964932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.1965195Z dist init r=1, world=2 2022-11-23T01:16:59.1965364Z dist init r=0, world=2 2022-11-23T01:16:59.1965575Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.1966014Z test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14486 2022-11-23T01:16:59.1966471Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14487 2022-11-23T01:16:59.1966921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1967254Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1967687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1968023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1968366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.1968836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1969226Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1969642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1969990Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1970327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.1970812Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1971336Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1971773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.1972124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.1972374Z dist init r=1, world=2 2022-11-23T01:16:59.1972559Z dist init r=0, world=2 2022-11-23T01:16:59.1972773Z skip: Need at least 2 CUDA devices (1.616s) 2022-11-23T01:16:59.1973202Z test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14514 2022-11-23T01:16:59.1973680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14515 2022-11-23T01:16:59.1974139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1974472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1974894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1975244Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1975584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.1976053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1976366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1976797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1977143Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1977465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.1977968Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1978493Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1978880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.1979218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.1979481Z dist init r=1, world=2 2022-11-23T01:16:59.1979664Z dist init r=0, world=2 2022-11-23T01:16:59.1979859Z skip: Need at least 2 CUDA devices (1.616s) 2022-11-23T01:16:59.1980297Z test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14542 2022-11-23T01:16:59.1980768Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14543 2022-11-23T01:16:59.1981229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1981545Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1982012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1982360Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1982699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.1983156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1983487Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1983912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1984293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1984631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.1985128Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1985649Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1986024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.1986375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.1986635Z dist init r=1, world=2 2022-11-23T01:16:59.1986802Z dist init r=0, world=2 2022-11-23T01:16:59.1987011Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.1987451Z test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14570 2022-11-23T01:16:59.1987926Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14571 2022-11-23T01:16:59.1988373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1988703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1989137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1989482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1989807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.1990275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1990606Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1991021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1991369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1991712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.1992300Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1992809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1993198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.1993548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.1993812Z dist init r=0, world=2 2022-11-23T01:16:59.1993983Z dist init r=1, world=2 2022-11-23T01:16:59.1994191Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.1994668Z test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14598 2022-11-23T01:16:59.1995121Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14599 2022-11-23T01:16:59.1995575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1995906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1996338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1996676Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1997049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.1997522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.1997839Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.1998267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.1998612Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.1998949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.1999434Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.1999954Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2000346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.2000999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.2001257Z dist init r=0, world=2 2022-11-23T01:16:59.2001442Z dist init r=1, world=2 2022-11-23T01:16:59.2001654Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.2001994Z test_summon_from_non_fsdp (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14626 2022-11-23T01:16:59.2002385Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14627 2022-11-23T01:16:59.2002849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2003167Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2003595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2003945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2004290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.2004748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2005075Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2005506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2005852Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2006178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.2006677Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2007202Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2007633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.2007986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.2008244Z dist init r=0, world=2 2022-11-23T01:16:59.2008428Z dist init r=1, world=2 2022-11-23T01:16:59.2008624Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.2009050Z test_summon_full_param_recursive_recurse_False_summon_outer_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14654 2022-11-23T01:16:59.2009511Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14655 2022-11-23T01:16:59.2009994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2010324Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2010766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2011101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2011434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.2011901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2012223Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2012639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2012986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2013316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.2013800Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2014321Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2014709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.2015055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.2015303Z dist init r=1, world=2 2022-11-23T01:16:59.2015480Z dist init r=0, world=2 2022-11-23T01:16:59.2015685Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.2016090Z test_summon_full_param_recursive_recurse_False_summon_outer_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14682 2022-11-23T01:16:59.2016545Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14683 2022-11-23T01:16:59.2016996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2017323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2017739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2018084Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2018415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.2018878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2019192Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2019621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2019961Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2020319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.2020814Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2021331Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2021834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.2022168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.2022457Z dist init r=0, world=2 2022-11-23T01:16:59.2022636Z dist init r=1, world=2 2022-11-23T01:16:59.2022829Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.2023245Z test_summon_full_param_recursive_recurse_False_summon_outer_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14710 2022-11-23T01:16:59.2023691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14711 2022-11-23T01:16:59.2024145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2024461Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2024891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2025231Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2025567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.2026024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2026350Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2026777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2027108Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2027438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.2027931Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2028443Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2028813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.2029159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.2029414Z dist init r=0, world=2 2022-11-23T01:16:59.2029578Z dist init r=1, world=2 2022-11-23T01:16:59.2029785Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.2030200Z test_summon_full_param_recursive_recurse_False_summon_outer_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14738 2022-11-23T01:16:59.2030645Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14739 2022-11-23T01:16:59.2031083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2031410Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2031837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2032257Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2032578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.2033086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2033410Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2033821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2034161Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2034492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.2034985Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2035528Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2035917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.2036262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.2036517Z dist init r=1, world=2 2022-11-23T01:16:59.2036684Z dist init r=0, world=2 2022-11-23T01:16:59.2036890Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.2037307Z test_summon_full_param_recursive_recurse_True_summon_outer_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14766 2022-11-23T01:16:59.2037743Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14767 2022-11-23T01:16:59.2038195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2038520Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2038949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2039279Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2039611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.2040079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2040392Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2040944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2041291Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2041628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.2042105Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2042622Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2043005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.2043349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.2043593Z dist init r=1, world=2 2022-11-23T01:16:59.2043774Z dist init r=0, world=2 2022-11-23T01:16:59.2043982Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.2044385Z test_summon_full_param_recursive_recurse_True_summon_outer_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14794 2022-11-23T01:16:59.2044834Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14795 2022-11-23T01:16:59.2045344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2045675Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2046088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2046432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2046764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.2047216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2047541Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2048009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2048351Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2048672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.2049164Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2049686Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2050069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.2050401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.2050655Z dist init r=1, world=2 2022-11-23T01:16:59.2050841Z dist init r=0, world=2 2022-11-23T01:16:59.2051037Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.2051455Z test_summon_full_param_recursive_recurse_True_summon_outer_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14822 2022-11-23T01:16:59.2051898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14823 2022-11-23T01:16:59.2052351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2052668Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2053095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2053443Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2053764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.2054231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2054557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2054983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2055312Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2055643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.2056132Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2056644Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2057020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.2057366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.2057624Z dist init r=0, world=2 2022-11-23T01:16:59.2057794Z dist init r=1, world=2 2022-11-23T01:16:59.2058036Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.2058451Z test_summon_full_param_recursive_recurse_True_summon_outer_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14850 2022-11-23T01:16:59.2058897Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14851 2022-11-23T01:16:59.2059338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2059664Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2060092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2060454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2060791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.2061267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2061595Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2062006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2062348Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2062679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.2063162Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2063680Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2064060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.2064409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.2064655Z dist init r=1, world=2 2022-11-23T01:16:59.2064835Z dist init r=0, world=2 2022-11-23T01:16:59.2065042Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.2065414Z test_summon_full_param_shard_value_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14878 2022-11-23T01:16:59.2065832Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14879 2022-11-23T01:16:59.2066282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2066614Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2067028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2067376Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2067711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.2068173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2068484Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2068909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2069251Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2069574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.2070066Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2070615Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2070999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.2071331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.2071584Z dist init r=1, world=2 2022-11-23T01:16:59.2071764Z dist init r=0, world=2 2022-11-23T01:16:59.2072041Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.2072431Z test_summon_full_param_shard_value_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14906 2022-11-23T01:16:59.2072904Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14907 2022-11-23T01:16:59.2073354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2073673Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2074096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2074440Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2074775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.2075253Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2075751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2076080Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2076492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2076836Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2077169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.2077662Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2078034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.2078381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.2078641Z dist init r=0, world=2 2022-11-23T01:16:59.2078807Z dist init r=1, world=2 2022-11-23T01:16:59.2079012Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.2079374Z test_summon_full_param_writeback (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14934 2022-11-23T01:16:59.2079768Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14935 2022-11-23T01:16:59.2080212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2080540Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2081130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2081463Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2081794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.2082257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2082584Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2082997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2083397Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2083731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.2084225Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2084730Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2085112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.2085458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.2085743Z dist init r=1, world=2 2022-11-23T01:16:59.2085924Z dist init r=0, world=2 2022-11-23T01:16:59.2086130Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:16:59.2086531Z test_summon_full_params_equivalence_rank0_only_False_offload_to_cpu_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14962 2022-11-23T01:16:59.2086955Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14963 2022-11-23T01:16:59.2087406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2087734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2088144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2088485Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2088823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.2089289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2089604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2090027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2090368Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2090696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.2091175Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2091691Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2092077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.2092408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.2092668Z dist init r=0, world=2 2022-11-23T01:16:59.2092851Z dist init r=1, world=2 2022-11-23T01:16:59.2093057Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.2093442Z test_summon_full_params_equivalence_rank0_only_False_offload_to_cpu_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14990 2022-11-23T01:16:59.2093871Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14991 2022-11-23T01:16:59.2094321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2094635Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2095061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2095405Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2095783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.2096238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2096564Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2096986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2097326Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2097645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.2098170Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2098687Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2099063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.2099407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.2099659Z dist init r=1, world=2 2022-11-23T01:16:59.2099839Z dist init r=0, world=2 2022-11-23T01:16:59.2100032Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.2100430Z test_summon_full_params_equivalence_rank0_only_True_offload_to_cpu_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15018 2022-11-23T01:16:59.2100863Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15019 2022-11-23T01:16:59.2101309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2101638Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2102066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2102408Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2102727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.2103189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2103516Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2103927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2104272Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2104607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.2105100Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2105606Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2105991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.2106337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.2106593Z dist init r=1, world=2 2022-11-23T01:16:59.2106760Z dist init r=0, world=2 2022-11-23T01:16:59.2106968Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.2107363Z test_summon_full_params_equivalence_rank0_only_True_offload_to_cpu_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15046 2022-11-23T01:16:59.2107783Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15047 2022-11-23T01:16:59.2108257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2108585Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2109009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2109337Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2109671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.2110135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2110473Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2110896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2111240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2111574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.2112124Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2112645Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2113032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.2113378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.2113628Z dist init r=1, world=2 2022-11-23T01:16:59.2113804Z dist init r=0, world=2 2022-11-23T01:16:59.2114009Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:16:59.2114382Z test_summon_full_params_respects_reshard_after_forward (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15074 2022-11-23T01:16:59.2114801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15075 2022-11-23T01:16:59.2115253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2115584Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2115998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2116343Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2116678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.2117132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2117454Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2117881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2118221Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2118540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.2119032Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2119550Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2119935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.2120268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.2120518Z dist init r=1, world=2 2022-11-23T01:16:59.2120850Z dist init r=0, world=2 2022-11-23T01:16:59.2121101Z skip: Need at least 2 CUDA devices (1.613s) 2022-11-23T01:16:59.2121461Z test_summon_single_param (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15102 2022-11-23T01:16:59.2121849Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15103 2022-11-23T01:16:59.2122303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2122621Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2123048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2123430Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2123750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.2124221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2124548Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2124971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2125300Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2125634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.2126127Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2126634Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2127018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.2127367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.2127624Z dist init r=0, world=2 2022-11-23T01:16:59.2127790Z dist init r=1, world=2 2022-11-23T01:16:59.2127994Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:16:59.2128244Z test_with_grads_core (__main__.TestSummonFullParams) 2022-11-23T01:16:59.2128602Z Tests the core usage of ``summon_full_params(with_grads=True)``. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15130 2022-11-23T01:16:59.2128990Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15131 2022-11-23T01:16:59.2129443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2129774Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2130191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2130536Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2130870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.2131323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2131646Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2132069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2132410Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2132735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.2133233Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2133775Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2134162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.2134495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.2134748Z dist init r=0, world=2 2022-11-23T01:16:59.2134927Z dist init r=1, world=2 2022-11-23T01:16:59.2135121Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:16:59.2135380Z test_with_grads_none_grads (__main__.TestSummonFullParams) 2022-11-23T01:16:59.2135873Z Tests that if all ranks' ``FlatParameter`` has ``None`` gradient, then ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15158 2022-11-23T01:16:59.2136295Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15159 2022-11-23T01:16:59.2136732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2137063Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2137486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2137824Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2138162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:16:59.2138629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2138956Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2139370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2139715Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2140049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.2140530Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2141046Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:16:59.2141428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.2141773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:16:59.2142019Z dist init r=0, world=2 2022-11-23T01:16:59.2142199Z dist init r=1, world=2 2022-11-23T01:16:59.2142406Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:16:59.2142767Z test_summon_full_param_writeback (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15186 2022-11-23T01:16:59.2143299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:16:59.2143626Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:16:59.2144052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:16:59.2144382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:16:59.2144712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:16:59.2145201Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T01:16:59.2145588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:16:59.2145833Z dist init r=0, world=1 2022-11-23T01:16:59.2146039Z skip: Need at least 2 CUDA devices (1.510s) 2022-11-23T01:16:59.2146176Z 2022-11-23T01:16:59.2146407Z ---------------------------------------------------------------------- 2022-11-23T01:16:59.2146637Z Ran 52 tests in 84.247s 2022-11-23T01:16:59.2146751Z 2022-11-23T01:16:59.2146824Z OK (skipped=52) 2022-11-23T01:16:59.2146930Z 2022-11-23T01:16:59.2147013Z Generating XML reports... 2022-11-23T01:16:59.2147493Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParams-20221123011534.xml 2022-11-23T01:16:59.2148135Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParamsNoShard-20221123011534.xml 2022-11-23T01:16:59.2148429Z 2022-11-23T01:16:59.2148836Z ##[endgroup] 2022-11-23T01:16:59.2149331Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_summon_full_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_summon_full_params_nfoc1_sm) 2022-11-23T01:16:59.2149613Z 2022-11-23T01:16:59.2149809Z Running distributed/fsdp/test_fsdp_optim_state ... [2022-11-23 01:16:59.172163] 2022-11-23T01:16:59.2150339Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_optim_state.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:16:59.172682] 2022-11-23T01:18:35.0917273Z 2022-11-23T01:18:35.0917840Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_optim_state 2022-11-23T01:18:35.0921101Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_optim_state (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_optim_state_7i0yqsld) 2022-11-23T01:18:35.0925042Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state 2022-11-23T01:18:35.0925443Z 2022-11-23T01:18:35.0925569Z Running tests... 2022-11-23T01:18:35.0926070Z ---------------------------------------------------------------------- 2022-11-23T01:18:35.0961770Z test_flatten_sharded_optim_state_dict_nested (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.0962831Z Tests :meth:`flatten_sharded_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15211 2022-11-23T01:18:35.0963456Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15212 2022-11-23T01:18:35.0964002Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15213 2022-11-23T01:18:35.0964536Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15214 2022-11-23T01:18:35.0965295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.0965815Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.0966622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.0967093Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.0967616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.0968400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.0968941Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.0969659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.0970205Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.0970739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.0971488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.0972014Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.0972996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.0973443Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.0973959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.0974750Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.0975586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.0976108Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.0976902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.0977435Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.0977979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.0978790Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.0979706Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.0980310Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.0980693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.0981039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.0981380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.0981710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.0981962Z dist init r=0, world=4 2022-11-23T01:18:35.0982142Z dist init r=2, world=4 2022-11-23T01:18:35.0982306Z dist init r=1, world=4 2022-11-23T01:18:35.0982538Z dist init r=3, world=4 2022-11-23T01:18:35.0982740Z skip: Need at least 2 CUDA devices (2.085s) 2022-11-23T01:18:35.0983013Z test_flatten_sharded_optim_state_dict_transformer (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.0983532Z Tests :meth:`flatten_sharded_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15267 2022-11-23T01:18:35.0983924Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15268 2022-11-23T01:18:35.0984253Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15269 2022-11-23T01:18:35.0984569Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15270 2022-11-23T01:18:35.0985021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.0985351Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.0985761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.0986103Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.0986435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.0986897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.0987209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.0987638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.0987982Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.0988315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.0988816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.0989148Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.0989572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.0989899Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.0990235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.0990701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.0991066Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.0991478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.0991824Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.0992248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.0992746Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.0993261Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.0993779Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.0994301Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.0994671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.0995018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.0995357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.0995702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.0995945Z dist init r=1, world=4 2022-11-23T01:18:35.0996121Z dist init r=3, world=4 2022-11-23T01:18:35.0996297Z dist init r=0, world=4 2022-11-23T01:18:35.0996458Z dist init r=2, world=4 2022-11-23T01:18:35.0996662Z skip: Need at least 2 CUDA devices (1.821s) 2022-11-23T01:18:35.0996923Z test_full_optim_state_dict_keys (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.0997277Z Tests that the parameter keys returned by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15323 2022-11-23T01:18:35.0997650Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15324 2022-11-23T01:18:35.0997979Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15325 2022-11-23T01:18:35.0998306Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15326 2022-11-23T01:18:35.0998741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.0999065Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.0999486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.0999829Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1000147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1000790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1001124Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1001629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1002757Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1003093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1003564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1003877Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1004306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1004742Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1005079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1005537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1005870Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1006304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1006639Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1006976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1007476Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1007996Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1008500Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1009090Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1009481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1009833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1010164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1010503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1010767Z dist init r=3, world=4 2022-11-23T01:18:35.1010934Z dist init r=2, world=4 2022-11-23T01:18:35.1011116Z dist init r=1, world=4 2022-11-23T01:18:35.1011296Z dist init r=0, world=4 2022-11-23T01:18:35.1011491Z skip: Need at least 2 CUDA devices (1.821s) 2022-11-23T01:18:35.1011767Z test_full_optim_state_dict_nested_invalid (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1012158Z Tests that :meth:`full_optim_state_dict` raises an error when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15379 2022-11-23T01:18:35.1012549Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15380 2022-11-23T01:18:35.1012866Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15381 2022-11-23T01:18:35.1013193Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15382 2022-11-23T01:18:35.1013653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1013978Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1014412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1014760Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1015134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1015592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1015924Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1016353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1016683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1017016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1017519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1017846Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1018263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1018613Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1018948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1019443Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1019926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1020253Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1020683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1021022Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1021361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1021855Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1022378Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1022881Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1023267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1023616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1023963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1024295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1024552Z dist init r=1, world=4 2022-11-23T01:18:35.1024741Z dist init r=2, world=4 2022-11-23T01:18:35.1024908Z dist init r=0, world=4 2022-11-23T01:18:35.1025087Z dist init r=3, world=4 2022-11-23T01:18:35.1025295Z skip: Need at least 2 CUDA devices (1.720s) 2022-11-23T01:18:35.1025539Z test_optim_input_warning (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1025921Z Tests that passing the ``optim_input`` argument into optimizer state ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15435 2022-11-23T01:18:35.1026318Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15436 2022-11-23T01:18:35.1026652Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15437 2022-11-23T01:18:35.1026965Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15438 2022-11-23T01:18:35.1027415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1027748Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1028192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1028542Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1028879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1029347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1029663Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1030088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1030463Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1030798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1031251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1031579Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1032008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1032422Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1032756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1033261Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1033788Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1034269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1034599Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1035029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1035386Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1035710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1036204Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1036726Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1037103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1037453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1037796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1038141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1038390Z dist init r=3, world=4 2022-11-23T01:18:35.1038572Z dist init r=1, world=4 2022-11-23T01:18:35.1038755Z dist init r=0, world=4 2022-11-23T01:18:35.1038925Z dist init r=2, world=4 2022-11-23T01:18:35.1039133Z skip: Need at least 2 CUDA devices (1.821s) 2022-11-23T01:18:35.1039529Z test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1040039Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15491 2022-11-23T01:18:35.1040460Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15492 2022-11-23T01:18:35.1040915Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15493 2022-11-23T01:18:35.1041245Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15494 2022-11-23T01:18:35.1041701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1042037Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1042466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1042821Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1043202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1043678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1044012Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1044427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1044773Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1045111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1045577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1045895Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1046330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1046678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1047015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1047466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1047794Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1048220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1048551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1048881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1049375Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1049903Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1050409Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1050924Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1051310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1051665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1052000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1052341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1052603Z dist init r=3, world=4 2022-11-23T01:18:35.1052774Z dist init r=0, world=4 2022-11-23T01:18:35.1052956Z dist init r=1, world=4 2022-11-23T01:18:35.1053134Z dist init r=2, world=4 2022-11-23T01:18:35.1053328Z skip: Need at least 2 CUDA devices (1.822s) 2022-11-23T01:18:35.1053777Z test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1054285Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15547 2022-11-23T01:18:35.1054683Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15548 2022-11-23T01:18:35.1054999Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15549 2022-11-23T01:18:35.1055321Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15550 2022-11-23T01:18:35.1055811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1056144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1056567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1056917Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1057256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1057707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1058035Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1058463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1058822Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1059147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1059616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1059945Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1060386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1060774Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1061112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1061666Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1062155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1062484Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1062915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1063262Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1063581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1064079Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1064602Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1065118Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1065493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1065848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1066228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1066564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1066826Z dist init r=2, world=4 2022-11-23T01:18:35.1067013Z dist init r=3, world=4 2022-11-23T01:18:35.1067180Z dist init r=1, world=4 2022-11-23T01:18:35.1067358Z dist init r=0, world=4 2022-11-23T01:18:35.1067568Z skip: Need at least 2 CUDA devices (1.821s) 2022-11-23T01:18:35.1067970Z test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1068495Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15603 2022-11-23T01:18:35.1068893Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15604 2022-11-23T01:18:35.1069227Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15605 2022-11-23T01:18:35.1069556Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15606 2022-11-23T01:18:35.1069999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1070329Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1070760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1071098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1071435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1071905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1072313Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1072739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1073087Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1073427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1073882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1074210Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1074642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1074991Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1075314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1075814Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1076312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1076640Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1077053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1077405Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1077740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1078225Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1078745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1079299Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1079692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1080028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1080365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1080839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1081108Z dist init r=1, world=4 2022-11-23T01:18:35.1081328Z dist init r=0, world=4 2022-11-23T01:18:35.1081506Z dist init r=3, world=4 2022-11-23T01:18:35.1081687Z dist init r=2, world=4 2022-11-23T01:18:35.1081883Z skip: Need at least 2 CUDA devices (1.720s) 2022-11-23T01:18:35.1082285Z test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1082792Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15659 2022-11-23T01:18:35.1083199Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15660 2022-11-23T01:18:35.1083515Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15661 2022-11-23T01:18:35.1083838Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15662 2022-11-23T01:18:35.1084307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1084628Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1085057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1085407Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1085744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1086201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1086530Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1086958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1087294Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1087635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1088104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1088435Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1088846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1089189Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1089523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1089986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1090297Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1090723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1091066Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1091384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1091921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1092445Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1092967Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1093468Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1093852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1094231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1094579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1094915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1095175Z dist init r=1, world=4 2022-11-23T01:18:35.1095359Z dist init r=2, world=4 2022-11-23T01:18:35.1095527Z dist init r=3, world=4 2022-11-23T01:18:35.1095709Z dist init r=0, world=4 2022-11-23T01:18:35.1095916Z skip: Need at least 2 CUDA devices (1.721s) 2022-11-23T01:18:35.1096299Z test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1096803Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15715 2022-11-23T01:18:35.1097200Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15716 2022-11-23T01:18:35.1097532Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15717 2022-11-23T01:18:35.1097854Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15718 2022-11-23T01:18:35.1098307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1098638Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1099072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1099406Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1099742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1100216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1100533Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1100965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1101309Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1101643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1102097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1102425Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1102851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1103183Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1103518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1103990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1104345Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1104762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1105106Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1105441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1105944Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1106453Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1106999Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1107524Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1107913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1108247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1108585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1108928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1109176Z dist init r=1, world=4 2022-11-23T01:18:35.1109358Z dist init r=3, world=4 2022-11-23T01:18:35.1109537Z dist init r=2, world=4 2022-11-23T01:18:35.1109703Z dist init r=0, world=4 2022-11-23T01:18:35.1109911Z skip: Need at least 2 CUDA devices (1.820s) 2022-11-23T01:18:35.1110304Z test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1110807Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15771 2022-11-23T01:18:35.1111192Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15772 2022-11-23T01:18:35.1111520Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15773 2022-11-23T01:18:35.1111848Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15774 2022-11-23T01:18:35.1112377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1112714Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1113142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1113496Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1113827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1114300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1114630Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1115060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1115390Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1115728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1116200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1116512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1116975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1117326Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1117661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1118119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1118447Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1118875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1119235Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1119570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1120071Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1120692Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1121206Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1121719Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1122111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1122458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1122795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1123136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1123402Z dist init r=3, world=4 2022-11-23T01:18:35.1123572Z dist init r=2, world=4 2022-11-23T01:18:35.1123753Z dist init r=0, world=4 2022-11-23T01:18:35.1123933Z dist init r=1, world=4 2022-11-23T01:18:35.1124125Z skip: Need at least 2 CUDA devices (1.921s) 2022-11-23T01:18:35.1124519Z test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1125020Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15827 2022-11-23T01:18:35.1125422Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15828 2022-11-23T01:18:35.1125741Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15829 2022-11-23T01:18:35.1126068Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15830 2022-11-23T01:18:35.1126526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1126854Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1127271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1127621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1127962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1128417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1128746Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1129173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1129572Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1129893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1130361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1130693Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1131124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1131454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1131790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1132306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1132626Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1133056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1133402Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1133736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1134217Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1134738Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1135258Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1135774Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1136151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1136503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1136846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1137181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1137442Z dist init r=0, world=4 2022-11-23T01:18:35.1137626Z dist init r=2, world=4 2022-11-23T01:18:35.1137804Z dist init r=3, world=4 2022-11-23T01:18:35.1137967Z dist init r=1, world=4 2022-11-23T01:18:35.1138173Z skip: Need at least 2 CUDA devices (1.821s) 2022-11-23T01:18:35.1138571Z test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1139059Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15883 2022-11-23T01:18:35.1139456Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15884 2022-11-23T01:18:35.1139792Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15885 2022-11-23T01:18:35.1140122Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15886 2022-11-23T01:18:35.1140562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1140894Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1141327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1141663Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1142003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1142523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1142850Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1143273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1143616Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1143954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1144423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1144763Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1145191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1145538Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1145860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1146324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1146650Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1147078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1147407Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1147742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1148239Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1148747Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1149271Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1149796Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1150183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1150514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1150863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1151204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1151468Z dist init r=2, world=4 2022-11-23T01:18:35.1151638Z dist init r=3, world=4 2022-11-23T01:18:35.1151821Z dist init r=0, world=4 2022-11-23T01:18:35.1152002Z dist init r=1, world=4 2022-11-23T01:18:35.1152278Z skip: Need at least 2 CUDA devices (1.821s) 2022-11-23T01:18:35.1152680Z test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1153183Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15939 2022-11-23T01:18:35.1153583Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15940 2022-11-23T01:18:35.1153902Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15941 2022-11-23T01:18:35.1154230Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15942 2022-11-23T01:18:35.1154701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1155058Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1155494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1155843Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1156182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1156639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1156968Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1157427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1157759Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1158098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1158568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1158896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1159316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1159661Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1160003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1160474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1160987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1161423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1161770Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1162093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1162590Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1163108Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1163632Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1164140Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1164531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1164881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1165224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1165556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1165814Z dist init r=3, world=4 2022-11-23T01:18:35.1166001Z dist init r=1, world=4 2022-11-23T01:18:35.1166168Z dist init r=0, world=4 2022-11-23T01:18:35.1166346Z dist init r=2, world=4 2022-11-23T01:18:35.1166555Z skip: Need at least 2 CUDA devices (1.721s) 2022-11-23T01:18:35.1166937Z test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1167444Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15995 2022-11-23T01:18:35.1167903Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15996 2022-11-23T01:18:35.1168237Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15997 2022-11-23T01:18:35.1168552Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15998 2022-11-23T01:18:35.1169009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1169348Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1169790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1170179Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1170520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1170993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1171308Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1171735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1172087Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1172428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1172889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1173228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1173655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1174004Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1174325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1174793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1175119Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1175529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1175871Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1176207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1176703Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1177221Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1177740Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1178255Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1178640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1178978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1179317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1179668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1179917Z dist init r=3, world=4 2022-11-23T01:18:35.1180104Z dist init r=1, world=4 2022-11-23T01:18:35.1180288Z dist init r=0, world=4 2022-11-23T01:18:35.1180452Z dist init r=2, world=4 2022-11-23T01:18:35.1180694Z skip: Need at least 2 CUDA devices (1.821s) 2022-11-23T01:18:35.1181094Z test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1181599Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16051 2022-11-23T01:18:35.1181987Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16052 2022-11-23T01:18:35.1182321Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16053 2022-11-23T01:18:35.1182679Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16054 2022-11-23T01:18:35.1183137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1183456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1183889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1184239Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1184565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1185030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1185361Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1185791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1186124Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1186458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1186933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1187249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1187682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1188026Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1188359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1188812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1189144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1189573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1189918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1190234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1190729Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1191246Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1191751Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1192341Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1192734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1193084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1193453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1193798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1194060Z dist init r=2, world=4 2022-11-23T01:18:35.1194243Z dist init r=0, world=4 2022-11-23T01:18:35.1194410Z dist init r=3, world=4 2022-11-23T01:18:35.1194588Z dist init r=1, world=4 2022-11-23T01:18:35.1194795Z skip: Need at least 2 CUDA devices (1.820s) 2022-11-23T01:18:35.1195177Z test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1195709Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16107 2022-11-23T01:18:35.1196113Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16108 2022-11-23T01:18:35.1196430Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16109 2022-11-23T01:18:35.1196754Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16110 2022-11-23T01:18:35.1197208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1197542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1197962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1198311Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1198651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1199121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1199436Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1199862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1200209Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1200530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1201154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1201484Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1201917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1202250Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1202589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1203086Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1203584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1203899Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1204328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1204674Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1204997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1205488Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1206106Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1206623Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1206995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1207344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1207693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1208035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1208319Z dist init r=3, world=4 2022-11-23T01:18:35.1208507Z dist init r=0, world=4 2022-11-23T01:18:35.1208688Z dist init r=1, world=4 2022-11-23T01:18:35.1208853Z dist init r=2, world=4 2022-11-23T01:18:35.1209061Z skip: Need at least 2 CUDA devices (1.721s) 2022-11-23T01:18:35.1209460Z test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1209954Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16163 2022-11-23T01:18:35.1210347Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16164 2022-11-23T01:18:35.1210680Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16165 2022-11-23T01:18:35.1211006Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16166 2022-11-23T01:18:35.1211449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1211779Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1212214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1212562Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1212886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1213355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1213684Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1214097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1214443Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1214780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1215252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1215569Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1215993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1216336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1216658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1217150Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1217674Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1218171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1218519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1218948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1219293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1219630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1220109Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1220627Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1221044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1221392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1221727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1222066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1222330Z dist init r=0, world=4 2022-11-23T01:18:35.1222500Z dist init r=1, world=4 2022-11-23T01:18:35.1222681Z dist init r=2, world=4 2022-11-23T01:18:35.1222859Z dist init r=3, world=4 2022-11-23T01:18:35.1223053Z skip: Need at least 2 CUDA devices (1.720s) 2022-11-23T01:18:35.1223451Z test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1223962Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16219 2022-11-23T01:18:35.1224361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16220 2022-11-23T01:18:35.1224678Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16221 2022-11-23T01:18:35.1224998Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16222 2022-11-23T01:18:35.1225456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1225773Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1226205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1226551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1226888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1227348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1227678Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1228108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1228454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1228774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1229244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1229575Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1230001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1230346Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1230662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1231199Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1231696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1232019Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1232512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1232852Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1233183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1233708Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1234222Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1234737Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1235122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1235459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1235796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1236134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1236392Z dist init r=2, world=4 2022-11-23T01:18:35.1236560Z dist init r=0, world=4 2022-11-23T01:18:35.1236735Z dist init r=3, world=4 2022-11-23T01:18:35.1236909Z dist init r=1, world=4 2022-11-23T01:18:35.1237100Z skip: Need at least 2 CUDA devices (1.721s) 2022-11-23T01:18:35.1237492Z test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1237992Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16275 2022-11-23T01:18:35.1238376Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16276 2022-11-23T01:18:35.1238703Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16277 2022-11-23T01:18:35.1239023Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16278 2022-11-23T01:18:35.1239474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1239794Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1240218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1240564Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1241045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1241520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1241844Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1242266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1242594Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1242931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1243392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1243779Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1244196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1244537Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1244869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1245348Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1245842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1246209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1246634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1246967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1247299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1247791Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1248307Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1248814Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1249196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1249543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1249885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1250213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1250467Z dist init r=3, world=4 2022-11-23T01:18:35.1250647Z dist init r=1, world=4 2022-11-23T01:18:35.1250809Z dist init r=2, world=4 2022-11-23T01:18:35.1250984Z dist init r=0, world=4 2022-11-23T01:18:35.1251188Z skip: Need at least 2 CUDA devices (1.722s) 2022-11-23T01:18:35.1251568Z test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1252068Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16331 2022-11-23T01:18:35.1252466Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16332 2022-11-23T01:18:35.1252790Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16333 2022-11-23T01:18:35.1253105Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16334 2022-11-23T01:18:35.1253555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1253883Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1254300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1254644Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1254978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1255450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1255762Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1256217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1256562Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1256895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1257354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1257677Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1258100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1258461Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1258789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1259260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1259584Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1259996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1260335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1260665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1261149Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1261663Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1262182Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1262697Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1263065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1263410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1263746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1264089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1264333Z dist init r=2, world=4 2022-11-23T01:18:35.1264511Z dist init r=3, world=4 2022-11-23T01:18:35.1264687Z dist init r=0, world=4 2022-11-23T01:18:35.1264851Z dist init r=1, world=4 2022-11-23T01:18:35.1265057Z skip: Need at least 2 CUDA devices (1.720s) 2022-11-23T01:18:35.1265413Z test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1265871Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16387 2022-11-23T01:18:35.1266244Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16388 2022-11-23T01:18:35.1266569Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16389 2022-11-23T01:18:35.1266892Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16390 2022-11-23T01:18:35.1267333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1267660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1268089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1268433Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1268788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1269257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1269581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1269994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1270335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1270670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1271166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1271479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1271903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1272335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1272665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1273116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1273442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1273866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1274196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1274529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1275022Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1275539Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1276040Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1276551Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1276934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1277277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1277611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1277945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1278201Z dist init r=3, world=4 2022-11-23T01:18:35.1278372Z dist init r=2, world=4 2022-11-23T01:18:35.1278549Z dist init r=0, world=4 2022-11-23T01:18:35.1278724Z dist init r=1, world=4 2022-11-23T01:18:35.1278917Z skip: Need at least 2 CUDA devices (1.721s) 2022-11-23T01:18:35.1279275Z test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1279730Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16443 2022-11-23T01:18:35.1280119Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16444 2022-11-23T01:18:35.1280437Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16445 2022-11-23T01:18:35.1280950Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16446 2022-11-23T01:18:35.1281415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1281802Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1282234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1282577Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1282910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1283364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1283687Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1284158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1284498Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1284821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1285287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1285615Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1286026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1286363Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1286697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1287166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1287476Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1287900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1288239Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1288558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1289047Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1289564Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1290080Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1290582Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1290962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1291313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1291655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1291984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1292244Z dist init r=3, world=4 2022-11-23T01:18:35.1292425Z dist init r=1, world=4 2022-11-23T01:18:35.1292588Z dist init r=0, world=4 2022-11-23T01:18:35.1292762Z dist init r=2, world=4 2022-11-23T01:18:35.1292967Z skip: Need at least 2 CUDA devices (1.821s) 2022-11-23T01:18:35.1293310Z test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1293765Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16499 2022-11-23T01:18:35.1294182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16500 2022-11-23T01:18:35.1294515Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16501 2022-11-23T01:18:35.1294830Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16502 2022-11-23T01:18:35.1295279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1295605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1296033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1296398Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1296731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1297202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1297511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1297933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1298278Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1298609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1299061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1299386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1299811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1300151Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1300470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1300960Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1301454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1301766Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1302195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1302543Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1302878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1303358Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1303872Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1304388Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1304769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1305101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1305440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1305779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1306027Z dist init r=0, world=4 2022-11-23T01:18:35.1306205Z dist init r=1, world=4 2022-11-23T01:18:35.1306380Z dist init r=2, world=4 2022-11-23T01:18:35.1306543Z dist init r=3, world=4 2022-11-23T01:18:35.1306781Z skip: Need at least 2 CUDA devices (1.821s) 2022-11-23T01:18:35.1307142Z test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1307599Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16555 2022-11-23T01:18:35.1307972Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16556 2022-11-23T01:18:35.1308300Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16557 2022-11-23T01:18:35.1308624Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16558 2022-11-23T01:18:35.1309104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1309420Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1309850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1310196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1310518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1310982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1311305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1311732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1312147Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1312484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1312952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1313262Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1313683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1314027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1314358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1314837Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1315329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1315654Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1316081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1316412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1316743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1317234Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1317737Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1318247Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1318633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1318976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1319364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1319704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1319964Z dist init r=2, world=4 2022-11-23T01:18:35.1320144Z dist init r=1, world=4 2022-11-23T01:18:35.1320308Z dist init r=0, world=4 2022-11-23T01:18:35.1320482Z dist init r=3, world=4 2022-11-23T01:18:35.1320859Z skip: Need at least 2 CUDA devices (1.721s) 2022-11-23T01:18:35.1321119Z test_rekey_optim_state_dict_to_names (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1321502Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16611 2022-11-23T01:18:35.1321944Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16612 2022-11-23T01:18:35.1322258Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16613 2022-11-23T01:18:35.1322576Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16614 2022-11-23T01:18:35.1323035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1323363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1323776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1324119Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1324455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1324908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1325240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1325664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1326005Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1326324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1326787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1327112Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1327532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1327857Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1328194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1328658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1328972Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1329399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1329739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1330070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1330550Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1331067Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1331584Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1332135Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1332506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1332852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1333196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1333525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1333780Z dist init r=3, world=4 2022-11-23T01:18:35.1333958Z dist init r=0, world=4 2022-11-23T01:18:35.1334122Z dist init r=2, world=4 2022-11-23T01:18:35.1334326Z dist init r=1, world=4 2022-11-23T01:18:35.1334530Z skip: Need at least 2 CUDA devices (1.720s) 2022-11-23T01:18:35.1334858Z test_save_load_without_0th_param_state_state_dict_type_StateDictType_FULL_STATE_DICT (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1335287Z Tests saving and loading an optim state dict for Adam optimizer (i.e. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16667 2022-11-23T01:18:35.1335681Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16668 2022-11-23T01:18:35.1336010Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16669 2022-11-23T01:18:35.1336319Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16670 2022-11-23T01:18:35.1336772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1337100Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1337530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1337863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1338195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1338663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1338987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1339400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1339741Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1340072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1340552Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1341049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1341373Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1341799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1342130Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1342465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1342928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1343248Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1343659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1344005Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1344336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1344844Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1345366Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1345885Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1346268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1346602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1346972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1347129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1347203Z dist init r=0, world=4 2022-11-23T01:18:35.1347274Z dist init r=3, world=4 2022-11-23T01:18:35.1347348Z dist init r=1, world=4 2022-11-23T01:18:35.1347408Z dist init r=2, world=4 2022-11-23T01:18:35.1347512Z skip: Need at least 2 CUDA devices (1.721s) 2022-11-23T01:18:35.1347712Z test_save_load_without_0th_param_state_state_dict_type_StateDictType_SHARDED_STATE_DICT (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1347941Z Tests saving and loading an optim state dict for Adam optimizer (i.e. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16723 2022-11-23T01:18:35.1348099Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16724 2022-11-23T01:18:35.1348249Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16725 2022-11-23T01:18:35.1348402Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16726 2022-11-23T01:18:35.1348675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1348786Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1349067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1349201Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1349376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1349646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1349767Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1350042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1350177Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1350340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1350606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1350727Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1350999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1351129Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1351301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1351566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1351688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1351961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1352185Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1352361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1352660Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1352955Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1353248Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1353574Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1353739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1353906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1354062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1354206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1354281Z dist init r=0, world=4 2022-11-23T01:18:35.1354355Z dist init r=1, world=4 2022-11-23T01:18:35.1354426Z dist init r=2, world=4 2022-11-23T01:18:35.1354497Z dist init r=3, world=4 2022-11-23T01:18:35.1354600Z skip: Need at least 2 CUDA devices (1.822s) 2022-11-23T01:18:35.1354762Z test_scatter_full_optim_state_dict_nested_halve_world_size (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1355087Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16779 2022-11-23T01:18:35.1355242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16780 2022-11-23T01:18:35.1355393Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16781 2022-11-23T01:18:35.1355540Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16782 2022-11-23T01:18:35.1355813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1355935Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1356216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1356350Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1356524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1356783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1356903Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1357182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1357314Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1357486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1357781Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1358053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1358175Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1358454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1358575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1358777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1359074Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1359349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1359470Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1359749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1359882Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1360092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1360387Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1360779Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1360943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1361107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1361265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1361421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1361495Z dist init r=1, world=4 2022-11-23T01:18:35.1361572Z dist init r=3, world=4 2022-11-23T01:18:35.1361642Z dist init r=2, world=4 2022-11-23T01:18:35.1361702Z dist init r=0, world=4 2022-11-23T01:18:35.1361804Z skip: Need at least 2 CUDA devices (1.724s) 2022-11-23T01:18:35.1362034Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1362367Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16835 2022-11-23T01:18:35.1362521Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16836 2022-11-23T01:18:35.1362670Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16837 2022-11-23T01:18:35.1362814Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16838 2022-11-23T01:18:35.1363086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1363200Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1363476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1363613Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1363790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1364057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1364178Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1364460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1364593Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1364759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1365028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1365150Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1365476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1365609Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1365784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1366081Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1366351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1366510Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1366773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1366904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1367080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1367376Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1367670Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1367963Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1368127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1368290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1368448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1368594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1368668Z dist init r=0, world=4 2022-11-23T01:18:35.1368739Z dist init r=3, world=4 2022-11-23T01:18:35.1368810Z dist init r=2, world=4 2022-11-23T01:18:35.1368881Z dist init r=1, world=4 2022-11-23T01:18:35.1368983Z skip: Need at least 2 CUDA devices (1.821s) 2022-11-23T01:18:35.1369212Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1369544Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16891 2022-11-23T01:18:35.1369689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16892 2022-11-23T01:18:35.1369838Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16893 2022-11-23T01:18:35.1369983Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16894 2022-11-23T01:18:35.1370256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1370376Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1370657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1370790Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1370965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1371222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1371346Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1371621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1371780Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1371954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1372221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1372341Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1372615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1372747Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1372968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1373267Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1373535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1373656Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1373934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1374067Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1374242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1374539Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1374836Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1375121Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1375288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1375449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1375607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1375763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1375837Z dist init r=0, world=4 2022-11-23T01:18:35.1375909Z dist init r=2, world=4 2022-11-23T01:18:35.1375979Z dist init r=3, world=4 2022-11-23T01:18:35.1376038Z dist init r=1, world=4 2022-11-23T01:18:35.1376142Z skip: Need at least 2 CUDA devices (1.722s) 2022-11-23T01:18:35.1376372Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1376703Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16947 2022-11-23T01:18:35.1376858Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16948 2022-11-23T01:18:35.1377007Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16949 2022-11-23T01:18:35.1377153Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16950 2022-11-23T01:18:35.1377428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1377537Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1377817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1377951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1378126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1378433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1378555Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1378831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1378963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1379128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1379421Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1379744Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1380017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1380138Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1380418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1380551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1380725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1380994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1381105Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1381381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1381513Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1381688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1381984Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1382278Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1382441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1382601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1382758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1382904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1382978Z dist init r=3, world=4 2022-11-23T01:18:35.1383050Z dist init r=1, world=4 2022-11-23T01:18:35.1383121Z dist init r=0, world=4 2022-11-23T01:18:35.1383194Z dist init r=2, world=4 2022-11-23T01:18:35.1383297Z skip: Need at least 2 CUDA devices (1.822s) 2022-11-23T01:18:35.1383523Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1383851Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17003 2022-11-23T01:18:35.1383995Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17004 2022-11-23T01:18:35.1384143Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17005 2022-11-23T01:18:35.1384294Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17006 2022-11-23T01:18:35.1384568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1384719Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1384999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1385133Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1385308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1385562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1385684Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1385960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1386124Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1386299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1386569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1386690Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1386965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1387096Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1387258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1387554Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1387850Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1388123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1388246Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1388527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1388660Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1388833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1389127Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1389414Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1389580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1389742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1389899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1390056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1390131Z dist init r=1, world=4 2022-11-23T01:18:35.1390203Z dist init r=3, world=4 2022-11-23T01:18:35.1390273Z dist init r=0, world=4 2022-11-23T01:18:35.1390331Z dist init r=2, world=4 2022-11-23T01:18:35.1390434Z skip: Need at least 2 CUDA devices (1.721s) 2022-11-23T01:18:35.1390660Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1390998Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17059 2022-11-23T01:18:35.1391153Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17060 2022-11-23T01:18:35.1391336Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17061 2022-11-23T01:18:35.1391485Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17062 2022-11-23T01:18:35.1391759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1391869Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1392215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1392351Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1392559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1392859Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1393133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1393254Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1393534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1393668Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1393831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1394098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1394223Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1394500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1394636Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1394811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1395077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1395196Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1395459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1413948Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1414232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1414605Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1414907Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1415211Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1415378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1415538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1415696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1415857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1415935Z dist init r=0, world=4 2022-11-23T01:18:35.1416009Z dist init r=1, world=4 2022-11-23T01:18:35.1416068Z dist init r=3, world=4 2022-11-23T01:18:35.1416138Z dist init r=2, world=4 2022-11-23T01:18:35.1416243Z skip: Need at least 2 CUDA devices (1.821s) 2022-11-23T01:18:35.1416587Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1416932Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17115 2022-11-23T01:18:35.1417093Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17116 2022-11-23T01:18:35.1417248Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17117 2022-11-23T01:18:35.1417402Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17118 2022-11-23T01:18:35.1417681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1417841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1418127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1418268Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1418446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1418721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1418844Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1419126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1419261Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1419428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1419700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1419825Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1420103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1420240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1420417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1420718Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1420988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1421112Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1421379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1421513Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1421690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1421988Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1422280Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1422568Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1422732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1422894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1423055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1423233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1423311Z dist init r=2, world=4 2022-11-23T01:18:35.1423383Z dist init r=0, world=4 2022-11-23T01:18:35.1423455Z dist init r=3, world=4 2022-11-23T01:18:35.1423527Z dist init r=1, world=4 2022-11-23T01:18:35.1423632Z skip: Need at least 2 CUDA devices (1.722s) 2022-11-23T01:18:35.1423859Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1424180Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17171 2022-11-23T01:18:35.1424368Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17172 2022-11-23T01:18:35.1424525Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17173 2022-11-23T01:18:35.1424680Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17174 2022-11-23T01:18:35.1424954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1425075Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1425357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1425491Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1425668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1425929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1426050Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1426331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1426464Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1426641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1426909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1427030Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1427307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1427440Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1427604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1427872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1427995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1428273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1428403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1428581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1428878Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1429173Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1429453Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1429776Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1429945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1430107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1430268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1430432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1430507Z dist init r=1, world=4 2022-11-23T01:18:35.1430578Z dist init r=2, world=4 2022-11-23T01:18:35.1430638Z dist init r=0, world=4 2022-11-23T01:18:35.1430740Z dist init r=3, world=4 2022-11-23T01:18:35.1430844Z skip: Need at least 2 CUDA devices (1.720s) 2022-11-23T01:18:35.1431068Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1431404Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17227 2022-11-23T01:18:35.1431558Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17228 2022-11-23T01:18:35.1431712Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17229 2022-11-23T01:18:35.1431863Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17230 2022-11-23T01:18:35.1432228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1432339Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1432628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1432763Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1432941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1433213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1433335Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1433615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1433749Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1433912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1434182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1434305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1434586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1434720Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1434895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1435165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1435286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1435563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1435685Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1435860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1436160Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1436509Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1436803Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1437096Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1437261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1437423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1437620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1437768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1437842Z dist init r=1, world=4 2022-11-23T01:18:35.1437920Z dist init r=3, world=4 2022-11-23T01:18:35.1437993Z dist init r=0, world=4 2022-11-23T01:18:35.1438064Z dist init r=2, world=4 2022-11-23T01:18:35.1438168Z skip: Need at least 2 CUDA devices (1.821s) 2022-11-23T01:18:35.1438323Z test_scatter_full_optim_state_dict_transformer (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1438631Z Tests :meth:`scatter_full_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17283 2022-11-23T01:18:35.1438787Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17284 2022-11-23T01:18:35.1438942Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17285 2022-11-23T01:18:35.1439096Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17286 2022-11-23T01:18:35.1439369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1439495Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1439779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1439914Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1440088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1440346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1440468Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1440878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1441018Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1441193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1441467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1441588Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1441868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1441989Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1442162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1442430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1442554Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1442830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1443012Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1443191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1443488Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1443785Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1444067Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1444402Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1444565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1444730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1444894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1445052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1445128Z dist init r=1, world=4 2022-11-23T01:18:35.1445201Z dist init r=0, world=4 2022-11-23T01:18:35.1445259Z dist init r=2, world=4 2022-11-23T01:18:35.1445332Z dist init r=3, world=4 2022-11-23T01:18:35.1445436Z skip: Need at least 2 CUDA devices (1.821s) 2022-11-23T01:18:35.1445598Z test_shard_full_optim_state_dict_nested_halve_world_size (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1445939Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17339 2022-11-23T01:18:35.1446094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17340 2022-11-23T01:18:35.1446248Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17341 2022-11-23T01:18:35.1446401Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17342 2022-11-23T01:18:35.1446662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1446783Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1447062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1447198Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1447375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1447649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1447771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1448053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1448185Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1448348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1448612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1448729Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1449002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1449133Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1449303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1449598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1449721Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1449986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1450117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1450292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1450587Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1450911Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1451201Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1451499Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1451661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1451824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1451989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1452138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1452212Z dist init r=2, world=4 2022-11-23T01:18:35.1452286Z dist init r=3, world=4 2022-11-23T01:18:35.1452359Z dist init r=0, world=4 2022-11-23T01:18:35.1452430Z dist init r=1, world=4 2022-11-23T01:18:35.1452533Z skip: Need at least 2 CUDA devices (1.720s) 2022-11-23T01:18:35.1452759Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1453087Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17395 2022-11-23T01:18:35.1453242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17396 2022-11-23T01:18:35.1453396Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17397 2022-11-23T01:18:35.1453547Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17398 2022-11-23T01:18:35.1453821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1453943Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1454227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1454363Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1454537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1454797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1454918Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1455201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1455335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1455511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1455779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1455899Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1456210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1456332Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1456507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1456777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1456899Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1457177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1457343Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1457518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1457816Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1458108Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1458393Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1458687Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1458852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1459019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1459181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1459340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1459411Z dist init r=0, world=4 2022-11-23T01:18:35.1459481Z dist init r=1, world=4 2022-11-23T01:18:35.1459541Z dist init r=3, world=4 2022-11-23T01:18:35.1459613Z dist init r=2, world=4 2022-11-23T01:18:35.1459714Z skip: Need at least 2 CUDA devices (1.720s) 2022-11-23T01:18:35.1459937Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1460280Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17451 2022-11-23T01:18:35.1460435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17452 2022-11-23T01:18:35.1460588Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17453 2022-11-23T01:18:35.1460740Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17454 2022-11-23T01:18:35.1461013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1461123Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1461403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1461538Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1461715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1461983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1462105Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1462381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1462539Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1462704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1462964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1463076Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1463349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1463477Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1463674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1463936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1464055Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1464329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1464448Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1464613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1464906Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1465194Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1465480Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1465773Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1465934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1466094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1466252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1466402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1466473Z dist init r=1, world=4 2022-11-23T01:18:35.1466540Z dist init r=2, world=4 2022-11-23T01:18:35.1466604Z dist init r=3, world=4 2022-11-23T01:18:35.1466673Z dist init r=0, world=4 2022-11-23T01:18:35.1466773Z skip: Need at least 2 CUDA devices (1.820s) 2022-11-23T01:18:35.1466993Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1467319Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17507 2022-11-23T01:18:35.1467469Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17508 2022-11-23T01:18:35.1467614Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17509 2022-11-23T01:18:35.1467762Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17510 2022-11-23T01:18:35.1468028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1468143Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1468420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1468551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1468725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1469010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1469134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1469414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1469547Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1469720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1469985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1470133Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1470404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1470528Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1470695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1470986Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1471248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1471362Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1471636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1471768Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1471938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1472316Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1472602Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1472891Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1473053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1473214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1473370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1473530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1473598Z dist init r=3, world=4 2022-11-23T01:18:35.1473663Z dist init r=2, world=4 2022-11-23T01:18:35.1473722Z dist init r=0, world=4 2022-11-23T01:18:35.1473796Z dist init r=1, world=4 2022-11-23T01:18:35.1473893Z skip: Need at least 2 CUDA devices (1.721s) 2022-11-23T01:18:35.1474109Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1474438Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17563 2022-11-23T01:18:35.1474587Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17564 2022-11-23T01:18:35.1474736Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17565 2022-11-23T01:18:35.1474883Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17566 2022-11-23T01:18:35.1475147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1475290Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1475569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1475695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1475863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1476131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1476247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1476518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1476685Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1476847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1477116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1477236Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1477507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1477639Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1477807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1478072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1478197Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1478473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1478596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1478771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1479065Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1479356Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1479647Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1479937Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1480093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1480255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1480410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1480554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1480759Z dist init r=2, world=4 2022-11-23T01:18:35.1480836Z dist init r=1, world=4 2022-11-23T01:18:35.1480901Z dist init r=0, world=4 2022-11-23T01:18:35.1480968Z dist init r=3, world=4 2022-11-23T01:18:35.1481065Z skip: Need at least 2 CUDA devices (1.821s) 2022-11-23T01:18:35.1481289Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1481620Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17619 2022-11-23T01:18:35.1481770Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17620 2022-11-23T01:18:35.1481983Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17621 2022-11-23T01:18:35.1482134Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17622 2022-11-23T01:18:35.1482406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1482523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1482802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1482931Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1483135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1483393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1483510Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1483785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1483915Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1484087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1484349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1484467Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1484745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1484871Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1485034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1485327Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1485589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1485704Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1485978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1486104Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1486270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1486564Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1486854Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1487136Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1487295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1487457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1487619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1487774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1487854Z dist init r=3, world=4 2022-11-23T01:18:35.1487925Z dist init r=1, world=4 2022-11-23T01:18:35.1487996Z dist init r=2, world=4 2022-11-23T01:18:35.1488055Z dist init r=0, world=4 2022-11-23T01:18:35.1488159Z skip: Need at least 2 CUDA devices (1.821s) 2022-11-23T01:18:35.1488443Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1488783Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17675 2022-11-23T01:18:35.1488938Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17676 2022-11-23T01:18:35.1489092Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17677 2022-11-23T01:18:35.1489243Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17678 2022-11-23T01:18:35.1489514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1489653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1489931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1490067Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1490244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1490514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1490637Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1490917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1491053Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1491219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1491489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1491612Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1491886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1492019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1492193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1492462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1492583Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1492863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1492985Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1493164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1493458Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1493757Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1494047Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1494342Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1494506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1494671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1494832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1495009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1495085Z dist init r=3, world=4 2022-11-23T01:18:35.1495158Z dist init r=0, world=4 2022-11-23T01:18:35.1495231Z dist init r=1, world=4 2022-11-23T01:18:35.1495303Z dist init r=2, world=4 2022-11-23T01:18:35.1495407Z skip: Need at least 2 CUDA devices (1.721s) 2022-11-23T01:18:35.1495628Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1495969Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17731 2022-11-23T01:18:35.1496138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17732 2022-11-23T01:18:35.1496289Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17733 2022-11-23T01:18:35.1496443Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17734 2022-11-23T01:18:35.1496716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1496837Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1497116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1497249Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1497425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1497686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1497804Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1498085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1498219Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1498394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1498662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1498783Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1499060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1499192Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1499357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1499624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1499746Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1500025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1500159Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1500333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1500629Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1500921Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1501212Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1501520Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1501684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1501845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1502010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1502170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1502244Z dist init r=1, world=4 2022-11-23T01:18:35.1502316Z dist init r=0, world=4 2022-11-23T01:18:35.1502388Z dist init r=3, world=4 2022-11-23T01:18:35.1502474Z dist init r=2, world=4 2022-11-23T01:18:35.1502577Z skip: Need at least 2 CUDA devices (1.824s) 2022-11-23T01:18:35.1502796Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1503135Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17787 2022-11-23T01:18:35.1503291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17788 2022-11-23T01:18:35.1503444Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17789 2022-11-23T01:18:35.1503595Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17790 2022-11-23T01:18:35.1503866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1503976Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1504255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1504389Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1504568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1504840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1504959Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1505237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1505370Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1505534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1505804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1505925Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1506204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1506336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1506509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1506776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1506897Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1507172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1507292Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1507468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1507763Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1508088Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1508384Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1508677Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1508839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1509003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1509204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1509349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1509423Z dist init r=3, world=4 2022-11-23T01:18:35.1509498Z dist init r=2, world=4 2022-11-23T01:18:35.1509570Z dist init r=1, world=4 2022-11-23T01:18:35.1509642Z dist init r=0, world=4 2022-11-23T01:18:35.1509744Z skip: Need at least 2 CUDA devices (1.821s) 2022-11-23T01:18:35.1509895Z test_shard_full_optim_state_dict_transformer (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1510202Z Tests :meth:`shard_full_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17843 2022-11-23T01:18:35.1510351Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17844 2022-11-23T01:18:35.1510497Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17845 2022-11-23T01:18:35.1510649Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17846 2022-11-23T01:18:35.1510921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1511045Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1511323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1511458Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1511632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1511889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1512010Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1512392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1512530Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1512707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1512977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1513100Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1513382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1513515Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1513679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1513978Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1514252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1514373Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1514685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1514821Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1514996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1515290Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1515586Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1515899Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1516062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1516227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1516387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1516461Z dist init r=0, world=4 2022-11-23T01:18:35.1516535Z dist init r=3, world=4 2022-11-23T01:18:35.1516692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1516765Z dist init r=1, world=4 2022-11-23T01:18:35.1516824Z dist init r=2, world=4 2022-11-23T01:18:35.1516929Z skip: Need at least 2 CUDA devices (1.721s) 2022-11-23T01:18:35.1517166Z test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_FULL_STATE_DICT_add_to_fsdp_module_False (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1517394Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17899 2022-11-23T01:18:35.1517548Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17900 2022-11-23T01:18:35.1517704Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17901 2022-11-23T01:18:35.1517854Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17902 2022-11-23T01:18:35.1518129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1518240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1518521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1518656Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1518834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1519106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1519230Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1519506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1519643Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1519805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1520073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1520196Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1520478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1520710Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1520887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1521233Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1521504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1521627Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1521894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1522027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1522238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1522534Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1522833Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1523126Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1523289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1523452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1523613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1523760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1523837Z dist init r=3, world=4 2022-11-23T01:18:35.1523910Z dist init r=0, world=4 2022-11-23T01:18:35.1523984Z dist init r=1, world=4 2022-11-23T01:18:35.1524056Z dist init r=2, world=4 2022-11-23T01:18:35.1524161Z skip: Need at least 2 CUDA devices (1.720s) 2022-11-23T01:18:35.1524399Z test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_FULL_STATE_DICT_add_to_fsdp_module_True (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1524625Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17955 2022-11-23T01:18:35.1524767Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17956 2022-11-23T01:18:35.1524919Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17957 2022-11-23T01:18:35.1525071Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17958 2022-11-23T01:18:35.1525345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1525473Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1525756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1525893Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1526068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1526325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1526447Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1526728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1526862Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1527038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1527304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1527467Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1527744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1527880Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1528041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1528340Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1528608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1528759Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1529040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1529176Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1529348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1529644Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1529941Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1530223Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1530387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1530550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1530711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1530875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1530952Z dist init r=2, world=4 2022-11-23T01:18:35.1531024Z dist init r=1, world=4 2022-11-23T01:18:35.1531096Z dist init r=0, world=4 2022-11-23T01:18:35.1531154Z dist init r=3, world=4 2022-11-23T01:18:35.1531256Z skip: Need at least 2 CUDA devices (1.720s) 2022-11-23T01:18:35.1531495Z test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_SHARDED_STATE_DICT_add_to_fsdp_module_False (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1531722Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18011 2022-11-23T01:18:35.1531880Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18012 2022-11-23T01:18:35.1532033Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18013 2022-11-23T01:18:35.1532188Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18014 2022-11-23T01:18:35.1532461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1532571Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1532852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1532985Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1533161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1533435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1533557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1533861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1533998Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1534175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1534430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1534551Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1534830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1534963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1535168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1535467Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1535738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1535861Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1536126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1536258Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1536433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1536731Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1537028Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1537326Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1537489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1537651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1537811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1537960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1538034Z dist init r=1, world=4 2022-11-23T01:18:35.1538106Z dist init r=3, world=4 2022-11-23T01:18:35.1538177Z dist init r=2, world=4 2022-11-23T01:18:35.1538250Z dist init r=0, world=4 2022-11-23T01:18:35.1538353Z skip: Need at least 2 CUDA devices (1.720s) 2022-11-23T01:18:35.1538589Z test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_SHARDED_STATE_DICT_add_to_fsdp_module_True (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1538818Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18067 2022-11-23T01:18:35.1538960Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18068 2022-11-23T01:18:35.1539113Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18069 2022-11-23T01:18:35.1539264Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18070 2022-11-23T01:18:35.1539539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1539665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1539947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1540082Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1540284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1540545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1540667Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1540944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1541078Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1541254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1541550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1541673Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1541950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1542084Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1542245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1542513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1542630Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1542910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1543049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1543223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1543521Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1543816Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1544105Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1544387Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1544551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1544714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1544879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1545039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1545114Z dist init r=3, world=4 2022-11-23T01:18:35.1545188Z dist init r=0, world=4 2022-11-23T01:18:35.1545259Z dist init r=2, world=4 2022-11-23T01:18:35.1545317Z dist init r=1, world=4 2022-11-23T01:18:35.1545420Z skip: Need at least 2 CUDA devices (1.821s) 2022-11-23T01:18:35.1545549Z test_use_orig_params_error (__main__.TestFSDPOptimState) 2022-11-23T01:18:35.1545779Z Tests that the optimizer state checkpointing APIs raise an error ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18123 2022-11-23T01:18:35.1545935Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18124 2022-11-23T01:18:35.1546088Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18125 2022-11-23T01:18:35.1546242Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18126 2022-11-23T01:18:35.1546517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1546658Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1546938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1547071Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1547249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:18:35.1547516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1547637Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1547914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1548078Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1548240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:18:35.1548511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1548632Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1548907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1549040Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1549214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:18:35.1549512Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1549810Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1550083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:18:35.1550193Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:18:35.1550473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:18:35.1550606Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:18:35.1550780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:18:35.1551075Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1551372Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:18:35.1551536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:18:35.1551700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:18:35.1551861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:18:35.1552008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:18:35.1552172Z dist init r=2, world=4 2022-11-23T01:18:35.1552247Z dist init r=0, world=4 2022-11-23T01:18:35.1552320Z dist init r=3, world=4 2022-11-23T01:18:35.1552390Z dist init r=1, world=4 2022-11-23T01:18:35.1552495Z skip: Need at least 2 CUDA devices (1.821s) 2022-11-23T01:18:35.1552504Z 2022-11-23T01:18:35.1552706Z ---------------------------------------------------------------------- 2022-11-23T01:18:35.1552773Z Ran 53 tests in 94.284s 2022-11-23T01:18:35.1552789Z 2022-11-23T01:18:35.1552849Z OK (skipped=53) 2022-11-23T01:18:35.1552853Z 2022-11-23T01:18:35.1552938Z Generating XML reports... 2022-11-23T01:18:35.1553317Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state/TEST-TestFSDPOptimState-20221123011700.xml 2022-11-23T01:18:35.1553323Z 2022-11-23T01:18:35.1553773Z ##[endgroup] 2022-11-23T01:18:35.1554144Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_optim_state (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_optim_state_7i0yqsld) 2022-11-23T01:18:35.1554149Z 2022-11-23T01:18:35.1554342Z Running distributed/fsdp/test_fsdp_core ... [2022-11-23 01:18:35.094666] 2022-11-23T01:18:35.1554700Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_core.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:18:35.095012] 2022-11-23T01:20:12.6646354Z 2022-11-23T01:20:12.6647280Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_core 2022-11-23T01:20:12.6649584Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_core (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_core_0s0_3j9n) 2022-11-23T01:20:12.6650530Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_core 2022-11-23T01:20:12.6650831Z 2022-11-23T01:20:12.6650934Z Running tests... 2022-11-23T01:20:12.6651401Z ---------------------------------------------------------------------- 2022-11-23T01:20:12.6671493Z test_pre_backward_hook_registration_after_state_dict (__main__.TestHooks) 2022-11-23T01:20:12.6672165Z Tests that FSDP pre-backward hooks are registered on forward pass ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18190 2022-11-23T01:20:12.6673220Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18191 2022-11-23T01:20:12.6673555Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18192 2022-11-23T01:20:12.6673884Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18193 2022-11-23T01:20:12.6674358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6674700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6675135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6675471Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6675811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.6676286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6676617Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6677033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6677379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6677721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.6678177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6678505Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6678931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6679273Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6679594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.6680091Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6680783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6681125Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6681692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6682042Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6682380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.6682868Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6683388Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6683961Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6684349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.6684686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.6685025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.6685369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.6685631Z dist init r=2, world=4 2022-11-23T01:20:12.6685799Z dist init r=1, world=4 2022-11-23T01:20:12.6685978Z dist init r=3, world=4 2022-11-23T01:20:12.6686156Z dist init r=0, world=4 2022-11-23T01:20:12.6686348Z skip: Need at least 2 CUDA devices (2.086s) 2022-11-23T01:20:12.6686622Z test_pre_backward_hook_registration_cuda_first_False (__main__.TestHooks) 2022-11-23T01:20:12.6687132Z Tests that FSDP pre-backward hooks are registered on forward pass ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18246 2022-11-23T01:20:12.6687521Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18247 2022-11-23T01:20:12.6687853Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18248 2022-11-23T01:20:12.6688183Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18249 2022-11-23T01:20:12.6688633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6688952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6689559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6690072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6690433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.6691193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6691716Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6692483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6692963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6693301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.6693772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6694104Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6694521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6694871Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6695209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.6695726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6696057Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6696488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6696831Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6697152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.6697651Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6698209Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6698732Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6699234Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6699621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.6699971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.6700305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.6700651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.6700908Z dist init r=3, world=4 2022-11-23T01:20:12.6701092Z dist init r=0, world=4 2022-11-23T01:20:12.6701257Z dist init r=2, world=4 2022-11-23T01:20:12.6701435Z dist init r=1, world=4 2022-11-23T01:20:12.6701643Z skip: Need at least 2 CUDA devices (1.720s) 2022-11-23T01:20:12.6701902Z test_pre_backward_hook_registration_cuda_first_True (__main__.TestHooks) 2022-11-23T01:20:12.6702410Z Tests that FSDP pre-backward hooks are registered on forward pass ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18302 2022-11-23T01:20:12.6702816Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18303 2022-11-23T01:20:12.6703131Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18304 2022-11-23T01:20:12.6703458Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18305 2022-11-23T01:20:12.6703908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6704243Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6704657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6705003Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6705345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.6705813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6706126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6706557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6706903Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6707223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.6707695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6708024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6708484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6708816Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6709150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.6709616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6709927Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6710355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6710731Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6711065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.6711548Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6712150Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6712672Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6713183Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6713552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.6713903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.6714251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.6714584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.6714845Z dist init r=1, world=4 2022-11-23T01:20:12.6715027Z dist init r=0, world=4 2022-11-23T01:20:12.6715205Z dist init r=2, world=4 2022-11-23T01:20:12.6715370Z dist init r=3, world=4 2022-11-23T01:20:12.6715582Z skip: Need at least 2 CUDA devices (1.720s) 2022-11-23T01:20:12.6715878Z test_register_functions_called_cuda_first_False_mixed_precision_False (__main__.TestHooks) 2022-11-23T01:20:12.6716273Z Tests that ``_register_{pre|post}_backward_hooks()`` are called ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18358 2022-11-23T01:20:12.6716670Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18359 2022-11-23T01:20:12.6717001Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18360 2022-11-23T01:20:12.6717329Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18361 2022-11-23T01:20:12.6717769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6718103Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6718532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6718867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6719205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.6719676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6720006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6720420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6720930Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6721342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.6721807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6722137Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6722567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6722913Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6723235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.6723749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6724085Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6724513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6724843Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6725180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.6725674Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6726181Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6726702Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6727222Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6727611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.6727948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.6728287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.6728630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.6728890Z dist init r=0, world=4 2022-11-23T01:20:12.6729060Z dist init r=2, world=4 2022-11-23T01:20:12.6729242Z dist init r=1, world=4 2022-11-23T01:20:12.6729419Z dist init r=3, world=4 2022-11-23T01:20:12.6729611Z skip: Need at least 2 CUDA devices (1.721s) 2022-11-23T01:20:12.6729902Z test_register_functions_called_cuda_first_False_mixed_precision_True (__main__.TestHooks) 2022-11-23T01:20:12.6730315Z Tests that ``_register_{pre|post}_backward_hooks()`` are called ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18414 2022-11-23T01:20:12.6730693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18415 2022-11-23T01:20:12.6731022Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18416 2022-11-23T01:20:12.6731348Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18417 2022-11-23T01:20:12.6731799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6732118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6732547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6732893Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6733219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.6733687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6734050Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6734478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6734807Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6735140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.6735607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6735933Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6736377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6736721Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6737060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.6737539Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6738035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6738362Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6738789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6739121Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6739457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.6739949Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6740470Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6740972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6741361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.6741710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.6742042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.6742385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.6742643Z dist init r=0, world=4 2022-11-23T01:20:12.6742828Z dist init r=1, world=4 2022-11-23T01:20:12.6742993Z dist init r=3, world=4 2022-11-23T01:20:12.6743170Z dist init r=2, world=4 2022-11-23T01:20:12.6743376Z skip: Need at least 2 CUDA devices (1.721s) 2022-11-23T01:20:12.6743656Z test_register_functions_called_cuda_first_True_mixed_precision_False (__main__.TestHooks) 2022-11-23T01:20:12.6744067Z Tests that ``_register_{pre|post}_backward_hooks()`` are called ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18470 2022-11-23T01:20:12.6744459Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18471 2022-11-23T01:20:12.6744774Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18472 2022-11-23T01:20:12.6745097Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18473 2022-11-23T01:20:12.6745545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6745879Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6746294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6746675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6747014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.6747484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6747801Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6748230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6748574Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6748948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.6749416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6749745Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6750172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6750501Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6750835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.6751301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6751613Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6752037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6752499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6752835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.6753326Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6753849Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6754365Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6754880Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6755249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.6755601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.6755946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.6756294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.6756541Z dist init r=3, world=4 2022-11-23T01:20:12.6756725Z dist init r=1, world=4 2022-11-23T01:20:12.6756905Z dist init r=0, world=4 2022-11-23T01:20:12.6757069Z dist init r=2, world=4 2022-11-23T01:20:12.6757281Z skip: Need at least 2 CUDA devices (1.720s) 2022-11-23T01:20:12.6757576Z test_register_functions_called_cuda_first_True_mixed_precision_True (__main__.TestHooks) 2022-11-23T01:20:12.6757969Z Tests that ``_register_{pre|post}_backward_hooks()`` are called ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18526 2022-11-23T01:20:12.6758365Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18527 2022-11-23T01:20:12.6758700Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18528 2022-11-23T01:20:12.6759027Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18529 2022-11-23T01:20:12.6759508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6759846Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6760274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6760758Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6761106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.6761579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6761965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6762381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6762729Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6763067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.6763519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6763847Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6764272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6764621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6764940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.6765417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6765733Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6766153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6766494Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6766826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.6767305Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6767819Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6768339Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6768847Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6769217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.6769563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.6769907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.6770247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.6770494Z dist init r=1, world=4 2022-11-23T01:20:12.6770674Z dist init r=2, world=4 2022-11-23T01:20:12.6770849Z dist init r=3, world=4 2022-11-23T01:20:12.6771011Z dist init r=0, world=4 2022-11-23T01:20:12.6771217Z skip: Need at least 2 CUDA devices (1.820s) 2022-11-23T01:20:12.6771486Z test_transformer_no_grad_mixed_precision_False (__main__.TestNoGrad) 2022-11-23T01:20:12.6771969Z Tests that for an FSDP-wrapped transformer model with shared ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18582 2022-11-23T01:20:12.6772409Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18583 2022-11-23T01:20:12.6772739Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18584 2022-11-23T01:20:12.6773064Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18585 2022-11-23T01:20:12.6773499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6773828Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6774255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6774622Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6774955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.6775420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6775744Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6776153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6776495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6776825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.6777274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6777598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6778026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6778365Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6778685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.6779149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6779476Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6779897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6780222Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6780553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.6781047Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6781558Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6782073Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6782587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6782968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.6783304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.6783642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.6783981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.6784241Z dist init r=0, world=4 2022-11-23T01:20:12.6784409Z dist init r=2, world=4 2022-11-23T01:20:12.6784584Z dist init r=1, world=4 2022-11-23T01:20:12.6784759Z dist init r=3, world=4 2022-11-23T01:20:12.6784989Z skip: Need at least 2 CUDA devices (1.720s) 2022-11-23T01:20:12.6785258Z test_transformer_no_grad_mixed_precision_True (__main__.TestNoGrad) 2022-11-23T01:20:12.6785748Z Tests that for an FSDP-wrapped transformer model with shared ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18638 2022-11-23T01:20:12.6786128Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18639 2022-11-23T01:20:12.6786454Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18640 2022-11-23T01:20:12.6786775Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18641 2022-11-23T01:20:12.6787224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6787571Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6788000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6788346Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6788668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.6789132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6789455Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6789882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6790215Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6790551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.6791045Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6791542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6791854Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6792363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6792708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6793029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.6793496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6793824Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6794246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6794575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6794904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.6795392Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6795904Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6796407Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6796789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.6797135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.6797465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.6797850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.6798108Z dist init r=2, world=4 2022-11-23T01:20:12.6798286Z dist init r=1, world=4 2022-11-23T01:20:12.6798449Z dist init r=0, world=4 2022-11-23T01:20:12.6798623Z dist init r=3, world=4 2022-11-23T01:20:12.6798827Z skip: Need at least 2 CUDA devices (1.721s) 2022-11-23T01:20:12.6799092Z test_param_change_after_init_mixed_precision_False (__main__.TestParamInit) 2022-11-23T01:20:12.6799607Z Tests that changing FSDP model parameter values in-place after FSDP ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18694 2022-11-23T01:20:12.6800012Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18695 2022-11-23T01:20:12.6800362Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18696 2022-11-23T01:20:12.6800866Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18697 2022-11-23T01:20:12.6801327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6801656Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6802069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6802415Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6802748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.6803199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6803524Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6803950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6804295Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6804619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.6805083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6805408Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6805828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6806157Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6806487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.6806951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6807262Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6807684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6808025Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6808353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.6808830Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6809345Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6809860Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6810374Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6810805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.6811164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.6811505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.6811835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.6812093Z dist init r=0, world=4 2022-11-23T01:20:12.6812274Z dist init r=3, world=4 2022-11-23T01:20:12.6812449Z dist init r=2, world=4 2022-11-23T01:20:12.6812612Z dist init r=1, world=4 2022-11-23T01:20:12.6812817Z skip: Need at least 2 CUDA devices (1.820s) 2022-11-23T01:20:12.6813133Z test_param_change_after_init_mixed_precision_True (__main__.TestParamInit) 2022-11-23T01:20:12.6813638Z Tests that changing FSDP model parameter values in-place after FSDP ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18750 2022-11-23T01:20:12.6814047Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18751 2022-11-23T01:20:12.6814375Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18752 2022-11-23T01:20:12.6814684Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18753 2022-11-23T01:20:12.6815130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6815458Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6815888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6816222Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6816556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.6817018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6817341Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6817751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6818094Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6818427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.6818878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6819204Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6819626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6819966Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6820286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.6820749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6821072Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6821480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6821819Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6822157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.6822652Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6823157Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6823702Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6824216Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6824599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.6824933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.6825269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.6825642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.6825900Z dist init r=0, world=4 2022-11-23T01:20:12.6826067Z dist init r=3, world=4 2022-11-23T01:20:12.6826244Z dist init r=2, world=4 2022-11-23T01:20:12.6826418Z dist init r=1, world=4 2022-11-23T01:20:12.6826611Z skip: Need at least 2 CUDA devices (1.820s) 2022-11-23T01:20:12.6826889Z test_delayed_optim_step_offload_false_no_shard (__main__.TestParityWithDDP) 2022-11-23T01:20:12.6827287Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18806 2022-11-23T01:20:12.6827668Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18807 2022-11-23T01:20:12.6827993Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18808 2022-11-23T01:20:12.6828318Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18809 2022-11-23T01:20:12.6828773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6829088Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6829515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6829864Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6830186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.6830651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6830979Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6831403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6831731Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6832146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.6832617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6832931Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6833354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6833696Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6834027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.6834509Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6835003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6835334Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6835758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6836128Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6836462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.6836954Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6837458Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6837967Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6838398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.6838743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.6839071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.6839409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.6839667Z dist init r=1, world=4 2022-11-23T01:20:12.6839845Z dist init r=2, world=4 2022-11-23T01:20:12.6840009Z dist init r=0, world=4 2022-11-23T01:20:12.6840183Z dist init r=3, world=4 2022-11-23T01:20:12.6840386Z skip: Need at least 2 CUDA devices (1.819s) 2022-11-23T01:20:12.6840788Z test_delayed_optim_step_offload_false_none (__main__.TestParityWithDDP) 2022-11-23T01:20:12.6841191Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18862 2022-11-23T01:20:12.6841590Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18863 2022-11-23T01:20:12.6841905Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18864 2022-11-23T01:20:12.6842230Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18865 2022-11-23T01:20:12.6842685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6843015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6843430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6843773Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6844105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.6844558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6844885Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6845306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6845648Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6845964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.6846431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6846757Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6847179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6847502Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6847836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.6848298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6848613Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6849094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6849437Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6849768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.6850250Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6850767Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6851327Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6851840Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6852207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.6852551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.6852890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.6853222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.6853479Z dist init r=0, world=4 2022-11-23T01:20:12.6853660Z dist init r=3, world=4 2022-11-23T01:20:12.6853834Z dist init r=1, world=4 2022-11-23T01:20:12.6853999Z dist init r=2, world=4 2022-11-23T01:20:12.6854204Z skip: Need at least 2 CUDA devices (1.820s) 2022-11-23T01:20:12.6854490Z test_delayed_optim_step_offload_false_shard_grad_op (__main__.TestParityWithDDP) 2022-11-23T01:20:12.6854883Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18918 2022-11-23T01:20:12.6855279Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18919 2022-11-23T01:20:12.6855605Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18920 2022-11-23T01:20:12.6855914Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18921 2022-11-23T01:20:12.6856362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6856690Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6857116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6857450Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6857781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.6858248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6858572Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6858984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6859322Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6859655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.6860102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6860427Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6860848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6861191Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6861542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.6862011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6862335Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6862742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6863078Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6863412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.6863935Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6864440Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6864953Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6865471Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6865851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.6866185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.6866523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.6866866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.6867110Z dist init r=2, world=4 2022-11-23T01:20:12.6867288Z dist init r=3, world=4 2022-11-23T01:20:12.6867466Z dist init r=0, world=4 2022-11-23T01:20:12.6867640Z dist init r=1, world=4 2022-11-23T01:20:12.6867833Z skip: Need at least 2 CUDA devices (1.819s) 2022-11-23T01:20:12.6868114Z test_delayed_optim_step_offload_true_no_shard (__main__.TestParityWithDDP) 2022-11-23T01:20:12.6868998Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82490 for platform(s) linux, rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s) 2022-11-23T01:20:12.6869679Z test_delayed_optim_step_offload_true_none (__main__.TestParityWithDDP) 2022-11-23T01:20:12.6870066Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18974 2022-11-23T01:20:12.6870459Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18975 2022-11-23T01:20:12.6870790Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18976 2022-11-23T01:20:12.6871115Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18977 2022-11-23T01:20:12.6871557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6871891Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6872399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6872730Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6873066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.6873532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6873856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6874312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6874655Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6874987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.6875453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6875762Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6876185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6876556Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6876875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.6877341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6877666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6878090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6878418Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6878747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.6879244Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6879765Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6880267Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6880933Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6881319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.6881653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.6881995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.6882334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.6882592Z dist init r=0, world=4 2022-11-23T01:20:12.6882759Z dist init r=2, world=4 2022-11-23T01:20:12.6882933Z dist init r=1, world=4 2022-11-23T01:20:12.6883106Z dist init r=3, world=4 2022-11-23T01:20:12.6883297Z skip: Need at least 2 CUDA devices (1.719s) 2022-11-23T01:20:12.6883582Z test_delayed_optim_step_offload_true_shard_grad_op (__main__.TestParityWithDDP) 2022-11-23T01:20:12.6883986Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19030 2022-11-23T01:20:12.6884378Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19031 2022-11-23T01:20:12.6884690Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19032 2022-11-23T01:20:12.6885008Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19033 2022-11-23T01:20:12.6885461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6885783Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6886209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6886555Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6886961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.6887419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6887745Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6888168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6888498Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6888829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.6889334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6889660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6890071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6890412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6890745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.6891208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6891515Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6891936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6892276Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6892594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.6893086Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6893605Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6894118Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6894613Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6894998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.6895346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.6895695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.6896026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.6896280Z dist init r=2, world=4 2022-11-23T01:20:12.6896459Z dist init r=3, world=4 2022-11-23T01:20:12.6896623Z dist init r=0, world=4 2022-11-23T01:20:12.6896796Z dist init r=1, world=4 2022-11-23T01:20:12.6896998Z skip: Need at least 2 CUDA devices (1.720s) 2022-11-23T01:20:12.6897271Z test_delayed_reduce_scatter_offload_false_no_shard (__main__.TestParityWithDDP) 2022-11-23T01:20:12.6897674Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19086 2022-11-23T01:20:12.6898068Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19087 2022-11-23T01:20:12.6898397Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19088 2022-11-23T01:20:12.6898708Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19089 2022-11-23T01:20:12.6899155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6899516Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6899933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6900277Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6900610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.6901075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6901388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6901846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6902189Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6902512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.6902975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6903298Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6903719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6904046Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6904378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.6904847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6905170Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6905582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6905923Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6906257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.6906739Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6907255Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6907773Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6908285Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6908656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.6909001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.6909339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.6909679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.6909923Z dist init r=0, world=4 2022-11-23T01:20:12.6910102Z dist init r=1, world=4 2022-11-23T01:20:12.6910276Z dist init r=2, world=4 2022-11-23T01:20:12.6910439Z dist init r=3, world=4 2022-11-23T01:20:12.6910642Z skip: Need at least 2 CUDA devices (1.819s) 2022-11-23T01:20:12.6910920Z test_delayed_reduce_scatter_offload_false_none (__main__.TestParityWithDDP) 2022-11-23T01:20:12.6911804Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82704 for platform(s) linux, rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s) 2022-11-23T01:20:12.6912501Z test_delayed_reduce_scatter_offload_false_shard_grad_op (__main__.TestParityWithDDP) 2022-11-23T01:20:12.6913362Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82398 for platform(s) linux, rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s) 2022-11-23T01:20:12.6913944Z test_delayed_reduce_scatter_offload_true_no_shard (__main__.TestParityWithDDP) 2022-11-23T01:20:12.6914383Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19142 2022-11-23T01:20:12.6914765Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19143 2022-11-23T01:20:12.6915092Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19144 2022-11-23T01:20:12.6915411Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19145 2022-11-23T01:20:12.6915862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6916181Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6916609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6916952Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6917274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.6917742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6918066Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6918488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6918819Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6919155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.6919616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6919941Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6920354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6921937Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6922273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.6922735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6923059Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6923485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6923827Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6924146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.6924646Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6925166Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6925760Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6926262Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6926645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.6926994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.6927324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.6927665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.6927977Z dist init r=1, world=4 2022-11-23T01:20:12.6928156Z dist init r=2, world=4 2022-11-23T01:20:12.6928320Z dist init r=3, world=4 2022-11-23T01:20:12.6928492Z dist init r=0, world=4 2022-11-23T01:20:12.6928696Z skip: Need at least 2 CUDA devices (1.819s) 2022-11-23T01:20:12.6928965Z test_delayed_reduce_scatter_offload_true_none (__main__.TestParityWithDDP) 2022-11-23T01:20:12.6929830Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82399 for platform(s) linux, rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s) 2022-11-23T01:20:12.6930439Z test_delayed_reduce_scatter_offload_true_shard_grad_op (__main__.TestParityWithDDP) 2022-11-23T01:20:12.6931295Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82403 for platform(s) linux, rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s) 2022-11-23T01:20:12.6931996Z test_mixture_of_experts_offload_false_no_shard_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19198 2022-11-23T01:20:12.6932402Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19199 2022-11-23T01:20:12.6932729Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19200 2022-11-23T01:20:12.6933053Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19201 2022-11-23T01:20:12.6933502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6933821Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6934250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6950663Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6951098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.6951635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6951981Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6952506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6952844Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6953181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.6953689Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6954277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6954599Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6955026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6955369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6955691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.6956158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6956490Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6956949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6957277Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6957614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.6958108Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6958622Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6959118Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6959500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.6959845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.6960175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.6960518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.6960980Z dist init r=3, world=4 2022-11-23T01:20:12.6961162Z dist init r=2, world=4 2022-11-23T01:20:12.6961326Z dist init r=1, world=4 2022-11-23T01:20:12.6961503Z dist init r=0, world=4 2022-11-23T01:20:12.6961708Z skip: Need at least 2 CUDA devices (1.819s) 2022-11-23T01:20:12.6962079Z test_mixture_of_experts_offload_false_none_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19254 2022-11-23T01:20:12.6962496Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19255 2022-11-23T01:20:12.6962829Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19256 2022-11-23T01:20:12.6963149Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19257 2022-11-23T01:20:12.6963608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6963937Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6964362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6964694Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6965030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.6965499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6965825Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6966238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6966583Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6966914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.6967432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6967760Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6968186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6968529Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6968850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.6969317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6969685Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6970102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6970446Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6970776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.6971267Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6971773Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6972290Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6972807Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6973188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.6973523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.6973867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.6974211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.6974456Z dist init r=2, world=4 2022-11-23T01:20:12.6974635Z dist init r=0, world=4 2022-11-23T01:20:12.6974810Z dist init r=1, world=4 2022-11-23T01:20:12.6974984Z dist init r=3, world=4 2022-11-23T01:20:12.6975173Z skip: Need at least 2 CUDA devices (1.820s) 2022-11-23T01:20:12.6975564Z test_mixture_of_experts_offload_false_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19310 2022-11-23T01:20:12.6975991Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19311 2022-11-23T01:20:12.6976308Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19312 2022-11-23T01:20:12.6976632Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19313 2022-11-23T01:20:12.6977082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6977412Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6977824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6978167Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6978502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.6978956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6979279Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6979738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6980083Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6980403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.6980867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6981193Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6981608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6981984Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6982315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.6982805Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6983290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6983616Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6984040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6984385Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6984702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.6985190Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6985710Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6986223Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6986592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.6986937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.6987279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.6987610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.6987864Z dist init r=3, world=4 2022-11-23T01:20:12.6988042Z dist init r=0, world=4 2022-11-23T01:20:12.6988206Z dist init r=2, world=4 2022-11-23T01:20:12.6988383Z dist init r=1, world=4 2022-11-23T01:20:12.6988585Z skip: Need at least 2 CUDA devices (1.820s) 2022-11-23T01:20:12.6988968Z test_mixture_of_experts_offload_true_no_shard_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19366 2022-11-23T01:20:12.6989373Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19367 2022-11-23T01:20:12.6989702Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19368 2022-11-23T01:20:12.6990029Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19369 2022-11-23T01:20:12.6990466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6990792Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6991219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6991564Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6991886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.6992497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6992828Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6993253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6993581Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6993912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.6994377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6994723Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6995145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6995485Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6995821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.6996301Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6996793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.6997116Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.6997523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.6997862Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.6998196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.6998689Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6999189Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.6999693Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7000077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.7000417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.7000873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.7001217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.7001475Z dist init r=3, world=4 2022-11-23T01:20:12.7001642Z dist init r=0, world=4 2022-11-23T01:20:12.7001822Z dist init r=2, world=4 2022-11-23T01:20:12.7002000Z dist init r=1, world=4 2022-11-23T01:20:12.7002194Z skip: Need at least 2 CUDA devices (1.919s) 2022-11-23T01:20:12.7002577Z test_mixture_of_experts_offload_true_none_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19422 2022-11-23T01:20:12.7002994Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19423 2022-11-23T01:20:12.7003325Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19424 2022-11-23T01:20:12.7003639Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19425 2022-11-23T01:20:12.7004090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7004420Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7004842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7005224Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7005563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.7006033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7006344Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7006771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7007116Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7007496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.7007949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7008275Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7008697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7009027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7009360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.7009853Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7010350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7010665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7011083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7011424Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7011756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.7012231Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7012743Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7013250Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7013628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.7013958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.7014296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.7014640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.7014884Z dist init r=3, world=4 2022-11-23T01:20:12.7015064Z dist init r=2, world=4 2022-11-23T01:20:12.7015241Z dist init r=1, world=4 2022-11-23T01:20:12.7015403Z dist init r=0, world=4 2022-11-23T01:20:12.7015607Z skip: Need at least 2 CUDA devices (1.720s) 2022-11-23T01:20:12.7015993Z test_mixture_of_experts_offload_true_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19478 2022-11-23T01:20:12.7016415Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19479 2022-11-23T01:20:12.7016736Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19480 2022-11-23T01:20:12.7017061Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19481 2022-11-23T01:20:12.7017509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7017853Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7018281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7018622Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7018954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.7019403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7019724Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7020195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7020537Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7020859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.7021323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7021647Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7022053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7022396Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7022729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.7023223Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7023700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7024026Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7024452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7024784Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7025113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.7025602Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7026115Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7026617Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7026997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.7027342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.7027685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.7028016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.7028267Z dist init r=1, world=4 2022-11-23T01:20:12.7028445Z dist init r=3, world=4 2022-11-23T01:20:12.7028608Z dist init r=0, world=4 2022-11-23T01:20:12.7028781Z dist init r=2, world=4 2022-11-23T01:20:12.7028985Z skip: Need at least 2 CUDA devices (1.720s) 2022-11-23T01:20:12.7029364Z test_mixture_of_experts_with_delay_before_free_offload_false_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19534 2022-11-23T01:20:12.7029795Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19535 2022-11-23T01:20:12.7030154Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19536 2022-11-23T01:20:12.7030482Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19537 2022-11-23T01:20:12.7030924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7031249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7031680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7032026Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7032416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.7032920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7033249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7033666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7034008Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7034342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.7034810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7035122Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7035547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7035897Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7036218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.7036682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7037011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7037431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7037760Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7038093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.7038585Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7039105Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7039609Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7040119Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7040501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.7041032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.7041366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.7041707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.7041965Z dist init r=3, world=4 2022-11-23T01:20:12.7042136Z dist init r=0, world=4 2022-11-23T01:20:12.7042315Z dist init r=2, world=4 2022-11-23T01:20:12.7042492Z dist init r=1, world=4 2022-11-23T01:20:12.7042683Z skip: Need at least 2 CUDA devices (1.819s) 2022-11-23T01:20:12.7043127Z test_mixture_of_experts_with_delay_before_free_offload_false_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19590 2022-11-23T01:20:12.7043551Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19591 2022-11-23T01:20:12.7043882Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19592 2022-11-23T01:20:12.7044199Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19593 2022-11-23T01:20:12.7044653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7044985Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7045438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7045784Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7046117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.7046586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7046899Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7047320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7047664Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7047995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.7048443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7048771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7049194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7049529Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7049862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.7050354Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7050848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7051161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7051582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7051926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7052248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.7052739Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7053255Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7053764Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7054132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.7054475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.7054817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.7055159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.7055403Z dist init r=0, world=4 2022-11-23T01:20:12.7055581Z dist init r=1, world=4 2022-11-23T01:20:12.7055786Z dist init r=3, world=4 2022-11-23T01:20:12.7055950Z dist init r=2, world=4 2022-11-23T01:20:12.7056154Z skip: Need at least 2 CUDA devices (1.720s) 2022-11-23T01:20:12.7056557Z test_mixture_of_experts_with_delay_before_free_offload_false_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19646 2022-11-23T01:20:12.7056986Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19647 2022-11-23T01:20:12.7057303Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19648 2022-11-23T01:20:12.7057625Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19649 2022-11-23T01:20:12.7058105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7058420Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7058847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7059188Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7059522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.7059977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7060301Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7060722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7061052Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7061382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.7061851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7062175Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7062588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7062932Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7063266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.7063759Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7064243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7064567Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7064991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7065321Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7065653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.7066142Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7066656Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7067157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7067539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.7067881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.7068072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.7068235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.7068297Z dist init r=0, world=4 2022-11-23T01:20:12.7068368Z dist init r=1, world=4 2022-11-23T01:20:12.7068439Z dist init r=2, world=4 2022-11-23T01:20:12.7068510Z dist init r=3, world=4 2022-11-23T01:20:12.7068613Z skip: Need at least 2 CUDA devices (1.820s) 2022-11-23T01:20:12.7068877Z test_mixture_of_experts_with_delay_before_free_offload_true_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19702 2022-11-23T01:20:12.7069063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19703 2022-11-23T01:20:12.7069215Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19704 2022-11-23T01:20:12.7069356Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19705 2022-11-23T01:20:12.7069633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7069757Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7070039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7070175Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7070352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.7070622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7070747Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7071016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7071151Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7071327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.7071598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7071719Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7071997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7072210Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7072389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.7072661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7072771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7073053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7073185Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7073363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.7073661Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7073953Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7074250Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7074539Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7074745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.7074893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.7075054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.7075216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.7075291Z dist init r=3, world=4 2022-11-23T01:20:12.7075364Z dist init r=2, world=4 2022-11-23T01:20:12.7075435Z dist init r=0, world=4 2022-11-23T01:20:12.7075505Z dist init r=1, world=4 2022-11-23T01:20:12.7075596Z skip: Need at least 2 CUDA devices (1.820s) 2022-11-23T01:20:12.7075887Z test_mixture_of_experts_with_delay_before_free_offload_true_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19758 2022-11-23T01:20:12.7076045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19759 2022-11-23T01:20:12.7076199Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19760 2022-11-23T01:20:12.7076351Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19761 2022-11-23T01:20:12.7076626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7076749Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7077028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7077163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7077330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.7077599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7077722Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7078001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7078133Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7078309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.7078578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7078700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7078979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7079099Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7079276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.7079545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7079667Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7079948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7080081Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7080255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.7080550Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7080965Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7081310Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7081602Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7081765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.7081924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.7082084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.7082245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.7082357Z dist init r=2, world=4 2022-11-23T01:20:12.7082429Z dist init r=3, world=4 2022-11-23T01:20:12.7082489Z dist init r=0, world=4 2022-11-23T01:20:12.7082560Z dist init r=1, world=4 2022-11-23T01:20:12.7082665Z skip: Need at least 2 CUDA devices (1.720s) 2022-11-23T01:20:12.7082939Z test_mixture_of_experts_with_delay_before_free_offload_true_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19814 2022-11-23T01:20:12.7083094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19815 2022-11-23T01:20:12.7083248Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19816 2022-11-23T01:20:12.7083400Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19817 2022-11-23T01:20:12.7083673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7083784Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7084065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7084199Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7084377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.7084648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7084770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7085050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7085183Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7085345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.7085615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7085736Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7086013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7086146Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7086320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.7086589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7086710Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7086987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7087110Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7087287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.7087609Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7087905Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7088197Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7088488Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7088652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.7088809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.7089018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.7089167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.7089243Z dist init r=1, world=4 2022-11-23T01:20:12.7089314Z dist init r=3, world=4 2022-11-23T01:20:12.7089384Z dist init r=0, world=4 2022-11-23T01:20:12.7089455Z dist init r=2, world=4 2022-11-23T01:20:12.7089558Z skip: Need at least 2 CUDA devices (1.720s) 2022-11-23T01:20:12.7089820Z test_nested_always_wrap_model_offload_false_no_shard_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19870 2022-11-23T01:20:12.7089962Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19871 2022-11-23T01:20:12.7090115Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19872 2022-11-23T01:20:12.7090272Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19873 2022-11-23T01:20:12.7090547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7090672Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7090949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7091084Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7091260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.7091532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7091642Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7091925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7092063Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7092240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.7092511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7092633Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7092909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7093043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7093205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.7093475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7093598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7093877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7094039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7094216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.7094513Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7094807Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7095102Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7095422Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7095572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.7095731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.7095895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.7096056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.7096129Z dist init r=0, world=4 2022-11-23T01:20:12.7096202Z dist init r=2, world=4 2022-11-23T01:20:12.7096273Z dist init r=3, world=4 2022-11-23T01:20:12.7096332Z dist init r=1, world=4 2022-11-23T01:20:12.7096435Z skip: Need at least 2 CUDA devices (1.819s) 2022-11-23T01:20:12.7096691Z test_nested_always_wrap_model_offload_false_none_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19926 2022-11-23T01:20:12.7096848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19927 2022-11-23T01:20:12.7096999Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19928 2022-11-23T01:20:12.7097153Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19929 2022-11-23T01:20:12.7097426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7097547Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7097827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7097949Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7098122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.7098394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7098515Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7098795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7098929Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7099102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.7099371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7099480Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7099757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7099888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7100063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.7100328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7100477Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7100755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7100886Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7101060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.7101345Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7101641Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7101962Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7102253Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7102416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.7102575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.7102734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.7102891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.7102964Z dist init r=3, world=4 2022-11-23T01:20:12.7103025Z dist init r=0, world=4 2022-11-23T01:20:12.7103099Z dist init r=1, world=4 2022-11-23T01:20:12.7103174Z dist init r=2, world=4 2022-11-23T01:20:12.7103277Z skip: Need at least 2 CUDA devices (1.820s) 2022-11-23T01:20:12.7103544Z test_nested_always_wrap_model_offload_false_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19982 2022-11-23T01:20:12.7103700Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19983 2022-11-23T01:20:12.7103854Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19984 2022-11-23T01:20:12.7103993Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19985 2022-11-23T01:20:12.7104261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7104381Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7104661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7104797Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7104974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.7105246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7105367Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7105642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7105763Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7105936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.7106206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7106329Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7106606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7106738Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7106938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.7107205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7107326Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7107590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7107720Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7107891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.7108221Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7108519Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7108808Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7109095Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7109258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.7109415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.7109566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.7109729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.7109803Z dist init r=1, world=4 2022-11-23T01:20:12.7109876Z dist init r=2, world=4 2022-11-23T01:20:12.7109945Z dist init r=0, world=4 2022-11-23T01:20:12.7110016Z dist init r=3, world=4 2022-11-23T01:20:12.7110120Z skip: Need at least 2 CUDA devices (1.719s) 2022-11-23T01:20:12.7110367Z test_nested_always_wrap_model_offload_true_no_shard_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20038 2022-11-23T01:20:12.7110523Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20039 2022-11-23T01:20:12.7110676Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20040 2022-11-23T01:20:12.7110830Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20041 2022-11-23T01:20:12.7111103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7111226Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7111507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7111642Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7111818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.7112154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7112282Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7112565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7112699Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7112878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.7113147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7113304Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7113587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7113707Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7113882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.7114180Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7114453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7114604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7114884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7115022Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7115195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.7115493Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7115777Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7116070Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7116234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.7116396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.7116557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.7116720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.7116794Z dist init r=3, world=4 2022-11-23T01:20:12.7116866Z dist init r=1, world=4 2022-11-23T01:20:12.7116925Z dist init r=0, world=4 2022-11-23T01:20:12.7116996Z dist init r=2, world=4 2022-11-23T01:20:12.7117098Z skip: Need at least 2 CUDA devices (1.719s) 2022-11-23T01:20:12.7117359Z test_nested_always_wrap_model_offload_true_none_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20094 2022-11-23T01:20:12.7117516Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20095 2022-11-23T01:20:12.7117670Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20096 2022-11-23T01:20:12.7117824Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20097 2022-11-23T01:20:12.7118100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7118209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7118487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7118620Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7118794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.7119064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7119186Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7119466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7119598Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7119804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.7120062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7120185Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7120462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7120705Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7120883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.7121204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7121326Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7121605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7121741Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7121906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.7122201Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7122497Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7122792Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7123084Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7123249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.7123409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.7123570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.7123730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.7123791Z dist init r=0, world=4 2022-11-23T01:20:12.7123863Z dist init r=1, world=4 2022-11-23T01:20:12.7123935Z dist init r=2, world=4 2022-11-23T01:20:12.7124006Z dist init r=3, world=4 2022-11-23T01:20:12.7124110Z skip: Need at least 2 CUDA devices (1.819s) 2022-11-23T01:20:12.7124379Z test_nested_always_wrap_model_offload_true_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20150 2022-11-23T01:20:12.7124534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20151 2022-11-23T01:20:12.7124677Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20152 2022-11-23T01:20:12.7124829Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20153 2022-11-23T01:20:12.7125100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7125222Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7125503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7125637Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7125813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.7126087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7126209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7126512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7126648Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7126821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.7127092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7127214Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7127489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7127651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7127827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.7128113Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7128384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7128504Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7128778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7128911Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7129084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.7129384Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7129683Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7129974Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7130126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.7130283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.7130442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.7130602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.7130676Z dist init r=2, world=4 2022-11-23T01:20:12.7130752Z dist init r=0, world=4 2022-11-23T01:20:12.7130823Z dist init r=3, world=4 2022-11-23T01:20:12.7130893Z dist init r=1, world=4 2022-11-23T01:20:12.7130984Z skip: Need at least 2 CUDA devices (1.819s) 2022-11-23T01:20:12.7131233Z test_nested_wrapped_model_offload_false_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20206 2022-11-23T01:20:12.7131387Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20207 2022-11-23T01:20:12.7131541Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20208 2022-11-23T01:20:12.7131692Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20209 2022-11-23T01:20:12.7131965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7132086Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7132367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7132489Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7132661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.7132960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7133083Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7133363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7133497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7133672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.7133941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7134108Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7134375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7134510Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7134683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.7134952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7135074Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7135351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7135484Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7135659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.7135944Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7136240Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7136535Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7136822Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7136988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.7137145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.7137307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.7137469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.7137542Z dist init r=0, world=4 2022-11-23T01:20:12.7137602Z dist init r=1, world=4 2022-11-23T01:20:12.7137674Z dist init r=2, world=4 2022-11-23T01:20:12.7137747Z dist init r=3, world=4 2022-11-23T01:20:12.7137850Z skip: Need at least 2 CUDA devices (1.719s) 2022-11-23T01:20:12.7138093Z test_nested_wrapped_model_offload_false_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20262 2022-11-23T01:20:12.7138247Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20263 2022-11-23T01:20:12.7138400Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20264 2022-11-23T01:20:12.7138553Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20265 2022-11-23T01:20:12.7138817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7138940Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7139253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7139387Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7139562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.7139832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7139957Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7140235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7140387Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7140564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.7140832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7140956Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7141232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7141366Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7141539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.7141808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7141930Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7142199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7142333Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7142512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.7142810Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7143105Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7143400Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7143689Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7143856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.7144014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.7144162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.7144324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.7144400Z dist init r=2, world=4 2022-11-23T01:20:12.7144471Z dist init r=1, world=4 2022-11-23T01:20:12.7144542Z dist init r=0, world=4 2022-11-23T01:20:12.7144613Z dist init r=3, world=4 2022-11-23T01:20:12.7144716Z skip: Need at least 2 CUDA devices (1.720s) 2022-11-23T01:20:12.7144954Z test_nested_wrapped_model_offload_false_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20318 2022-11-23T01:20:12.7145108Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20319 2022-11-23T01:20:12.7145263Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20320 2022-11-23T01:20:12.7145414Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20321 2022-11-23T01:20:12.7145715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7145838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7146119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7146253Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7146428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.7146686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7146839Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7147125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7147259Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7147434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.7147701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7147825Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7148101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7148233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7148396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.7148667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7148788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7149069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7149202Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7149376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.7149670Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7149964Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7150257Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7150536Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7150699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.7150856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.7151016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.7151176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.7151249Z dist init r=3, world=4 2022-11-23T01:20:12.7151320Z dist init r=2, world=4 2022-11-23T01:20:12.7151390Z dist init r=1, world=4 2022-11-23T01:20:12.7151449Z dist init r=0, world=4 2022-11-23T01:20:12.7151552Z skip: Need at least 2 CUDA devices (1.719s) 2022-11-23T01:20:12.7151802Z test_nested_wrapped_model_offload_true_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20374 2022-11-23T01:20:12.7151957Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20375 2022-11-23T01:20:12.7152218Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20376 2022-11-23T01:20:12.7152372Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20377 2022-11-23T01:20:12.7152647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7152770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7153039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7153175Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7153416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.7153688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7153809Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7154090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7154225Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7154401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.7154658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7154782Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7155059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7155198Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7155373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.7155643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7155766Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7156042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7156175Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7156337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.7156632Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7156932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7157230Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7157520Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7157683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.7157844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.7158003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.7158163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.7158227Z dist init r=1, world=4 2022-11-23T01:20:12.7158299Z dist init r=3, world=4 2022-11-23T01:20:12.7158371Z dist init r=0, world=4 2022-11-23T01:20:12.7158442Z dist init r=2, world=4 2022-11-23T01:20:12.7158544Z skip: Need at least 2 CUDA devices (1.720s) 2022-11-23T01:20:12.7158814Z test_nested_wrapped_model_offload_true_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20430 2022-11-23T01:20:12.7158972Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20431 2022-11-23T01:20:12.7159112Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20432 2022-11-23T01:20:12.7159264Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20433 2022-11-23T01:20:12.7159535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7159657Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7159966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7160100Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7160277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.7160549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7160767Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7161034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7161168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7161343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.7161607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7161731Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7162008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7162142Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7162317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.7162613Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7162873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7162994Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7163272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7163407Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7163583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.7163880Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7164173Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7164462Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7164625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.7164770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.7164931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.7165088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.7165156Z dist init r=2, world=4 2022-11-23T01:20:12.7165225Z dist init r=1, world=4 2022-11-23T01:20:12.7165340Z dist init r=0, world=4 2022-11-23T01:20:12.7165411Z dist init r=3, world=4 2022-11-23T01:20:12.7165503Z skip: Need at least 2 CUDA devices (1.719s) 2022-11-23T01:20:12.7165747Z test_nested_wrapped_model_offload_true_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20486 2022-11-23T01:20:12.7165899Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20487 2022-11-23T01:20:12.7166050Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20488 2022-11-23T01:20:12.7166196Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20489 2022-11-23T01:20:12.7166502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7166619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7166895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7167024Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7167189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.7167453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7167569Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7167844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7167979Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7168150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.7168416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7168532Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7168797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7168925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7169098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.7169361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7169477Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7169753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7169881Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7170052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.7170342Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7170626Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7170917Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7171201Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7171362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.7171517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.7171702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.7171861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.7171930Z dist init r=2, world=4 2022-11-23T01:20:12.7171989Z dist init r=1, world=4 2022-11-23T01:20:12.7172055Z dist init r=3, world=4 2022-11-23T01:20:12.7172123Z dist init r=0, world=4 2022-11-23T01:20:12.7172223Z skip: Need at least 2 CUDA devices (1.719s) 2022-11-23T01:20:12.7172501Z test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20542 2022-11-23T01:20:12.7172681Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20543 2022-11-23T01:20:12.7172827Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20544 2022-11-23T01:20:12.7172972Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20545 2022-11-23T01:20:12.7173235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7173354Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7173629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7173760Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7173929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.7174196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7174315Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7174591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7174719Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7174884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.7175151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7175268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7175544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7175673Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7175844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.7176108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7176223Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7176490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7176617Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7176787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.7177080Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7177366Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7177660Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7177948Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7178142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.7178298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.7178452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.7178601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.7178674Z dist init r=1, world=4 2022-11-23T01:20:12.7178745Z dist init r=2, world=4 2022-11-23T01:20:12.7178815Z dist init r=3, world=4 2022-11-23T01:20:12.7178885Z dist init r=0, world=4 2022-11-23T01:20:12.7179027Z skip: Need at least 2 CUDA devices (1.719s) 2022-11-23T01:20:12.7179302Z test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20598 2022-11-23T01:20:12.7179447Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20599 2022-11-23T01:20:12.7179595Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20600 2022-11-23T01:20:12.7179747Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20601 2022-11-23T01:20:12.7180022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7180144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7180425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7180558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7180735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.7181004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7181115Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7181394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7181527Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7181703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.7181973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7182094Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7182376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7182509Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7182674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.7182943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7183065Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7183340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7183475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7183649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.7183949Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7184241Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7184561Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7184840Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7185002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.7185159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.7185319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.7185478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.7185584Z dist init r=2, world=4 2022-11-23T01:20:12.7185655Z dist init r=3, world=4 2022-11-23T01:20:12.7185724Z dist init r=1, world=4 2022-11-23T01:20:12.7185782Z dist init r=0, world=4 2022-11-23T01:20:12.7185885Z skip: Need at least 2 CUDA devices (1.819s) 2022-11-23T01:20:12.7186176Z test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20654 2022-11-23T01:20:12.7186331Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20655 2022-11-23T01:20:12.7186483Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20656 2022-11-23T01:20:12.7186633Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20657 2022-11-23T01:20:12.7186906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7187028Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7187296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7187432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7187606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.7187876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7187998Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7188274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7188407Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7188581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.7188878Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7189136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7189256Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7189534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7189664Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7189839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.7190108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7190230Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7190506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7190638Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7190830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.7191128Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7191418Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7191705Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7191868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.7192140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.7192311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.7192471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.7192546Z dist init r=3, world=4 2022-11-23T01:20:12.7192606Z dist init r=2, world=4 2022-11-23T01:20:12.7192677Z dist init r=0, world=4 2022-11-23T01:20:12.7192750Z dist init r=1, world=4 2022-11-23T01:20:12.7192853Z skip: Need at least 2 CUDA devices (1.720s) 2022-11-23T01:20:12.7193133Z test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20710 2022-11-23T01:20:12.7193288Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20711 2022-11-23T01:20:12.7193442Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20712 2022-11-23T01:20:12.7193583Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20713 2022-11-23T01:20:12.7193859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7193981Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7194263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7194396Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7194571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.7194839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7194959Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7195239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7195358Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7195533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.7195798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7195923Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7196200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7196332Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7196503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.7196799Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7197065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7197171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7197474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7197607Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7197779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.7198072Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7198365Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7198683Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7198845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.7198998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.7199159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.7199318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.7199392Z dist init r=1, world=4 2022-11-23T01:20:12.7199463Z dist init r=3, world=4 2022-11-23T01:20:12.7199536Z dist init r=0, world=4 2022-11-23T01:20:12.7199607Z dist init r=2, world=4 2022-11-23T01:20:12.7199711Z skip: Need at least 2 CUDA devices (1.720s) 2022-11-23T01:20:12.7199974Z test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20766 2022-11-23T01:20:12.7200132Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20767 2022-11-23T01:20:12.7200283Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20768 2022-11-23T01:20:12.7200438Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20769 2022-11-23T01:20:12.7200844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7200968Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7201251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7201385Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7201548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.7201822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7201944Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7202228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7202363Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7202543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.7202814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7202936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7203217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7203340Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7203516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.7203859Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7204130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7204253Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7204529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7204661Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7204835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.7205165Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7205451Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7205743Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7205905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.7206070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.7206229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.7206390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.7206463Z dist init r=0, world=4 2022-11-23T01:20:12.7206535Z dist init r=2, world=4 2022-11-23T01:20:12.7206597Z dist init r=3, world=4 2022-11-23T01:20:12.7206668Z dist init r=1, world=4 2022-11-23T01:20:12.7206771Z skip: Need at least 2 CUDA devices (1.820s) 2022-11-23T01:20:12.7207057Z test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20822 2022-11-23T01:20:12.7207214Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20823 2022-11-23T01:20:12.7207366Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20824 2022-11-23T01:20:12.7207521Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20825 2022-11-23T01:20:12.7207794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7207903Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7208182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7208316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7208492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.7208764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7208887Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7209166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7209298Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7209474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.7209761Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7210030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7210151Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7210459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7210593Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7210766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.7211035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7211155Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7211434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7211584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7211757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.7212057Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7212351Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7212638Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7212802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.7212964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.7213127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.7213285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.7213346Z dist init r=0, world=4 2022-11-23T01:20:12.7213418Z dist init r=1, world=4 2022-11-23T01:20:12.7213491Z dist init r=3, world=4 2022-11-23T01:20:12.7213561Z dist init r=2, world=4 2022-11-23T01:20:12.7213665Z skip: Need at least 2 CUDA devices (1.720s) 2022-11-23T01:20:12.7213915Z test_transformer_offload_false_no_shard_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20878 2022-11-23T01:20:12.7214069Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20879 2022-11-23T01:20:12.7214210Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20880 2022-11-23T01:20:12.7214362Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20881 2022-11-23T01:20:12.7214638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7214759Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7215038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7215172Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7215346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.7215613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7215735Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7216000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7216135Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7216310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.7216603Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7216902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7217025Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7217301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7217433Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7217595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.7217862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7218010Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7218285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7218421Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7218596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.7218891Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7219185Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7219471Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7219623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.7219784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.7219944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.7220103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.7220175Z dist init r=3, world=4 2022-11-23T01:20:12.7220247Z dist init r=2, world=4 2022-11-23T01:20:12.7220318Z dist init r=1, world=4 2022-11-23T01:20:12.7220388Z dist init r=0, world=4 2022-11-23T01:20:12.7220478Z skip: Need at least 2 CUDA devices (1.820s) 2022-11-23T01:20:12.7220724Z test_transformer_offload_false_none_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20934 2022-11-23T01:20:12.7220878Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20935 2022-11-23T01:20:12.7221033Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20936 2022-11-23T01:20:12.7221184Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20937 2022-11-23T01:20:12.7221459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7221581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7221858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7221980Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7222155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.7222423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7222547Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7222824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7222956Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7223165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.7223436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7223555Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7223818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7223949Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7224122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.7224465Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7224731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7224854Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7225127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7225260Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7225421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.7225716Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7226015Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7226304Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7226469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.7226632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.7226794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.7226955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.7227027Z dist init r=1, world=4 2022-11-23T01:20:12.7227087Z dist init r=3, world=4 2022-11-23T01:20:12.7227158Z dist init r=2, world=4 2022-11-23T01:20:12.7227228Z dist init r=0, world=4 2022-11-23T01:20:12.7227334Z skip: Need at least 2 CUDA devices (1.820s) 2022-11-23T01:20:12.7227591Z test_transformer_offload_false_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20990 2022-11-23T01:20:12.7227744Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20991 2022-11-23T01:20:12.7227899Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20992 2022-11-23T01:20:12.7228051Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20993 2022-11-23T01:20:12.7228308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7228428Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7228708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7228840Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7229015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.7229283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7229430Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7229711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7229832Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7230006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.7230272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7230397Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7230787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7230939Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7231113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.7231382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7231503Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7231779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7231911Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7232161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.7232466Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7232765Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7233052Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7233344Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7233508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.7233670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.7233832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.7233991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.7234067Z dist init r=0, world=4 2022-11-23T01:20:12.7234138Z dist init r=3, world=4 2022-11-23T01:20:12.7234197Z dist init r=2, world=4 2022-11-23T01:20:12.7234269Z dist init r=1, world=4 2022-11-23T01:20:12.7234372Z skip: Need at least 2 CUDA devices (1.720s) 2022-11-23T01:20:12.7234622Z test_transformer_offload_true_no_shard_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21046 2022-11-23T01:20:12.7234777Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21047 2022-11-23T01:20:12.7234930Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21048 2022-11-23T01:20:12.7235080Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21049 2022-11-23T01:20:12.7235355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7235464Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7235747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7235880Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7236095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.7236367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7236488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7236764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7236897Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7237069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.7237354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7237473Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7237750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7237881Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7238055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.7238320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7238442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7238720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7238839Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7239015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.7239308Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7239604Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7239890Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7240179Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7240342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.7240504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.7240784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.7240932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.7241006Z dist init r=1, world=4 2022-11-23T01:20:12.7241082Z dist init r=3, world=4 2022-11-23T01:20:12.7241155Z dist init r=2, world=4 2022-11-23T01:20:12.7241226Z dist init r=0, world=4 2022-11-23T01:20:12.7241329Z skip: Need at least 2 CUDA devices (1.719s) 2022-11-23T01:20:12.7241575Z test_transformer_offload_true_none_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21102 2022-11-23T01:20:12.7241731Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21103 2022-11-23T01:20:12.7241870Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21104 2022-11-23T01:20:12.7242021Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21105 2022-11-23T01:20:12.7242299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7242421Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7242750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7242890Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7243065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.7243335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7243443Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7243720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7243899Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7244071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.7244346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7244465Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7244744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7244876Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7245048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.7245302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7245424Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7245699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7245831Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7246007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.7246301Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7246598Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7246884Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7247172Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7247326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.7247487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.7247650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.7247810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.7247883Z dist init r=1, world=4 2022-11-23T01:20:12.7247955Z dist init r=0, world=4 2022-11-23T01:20:12.7248026Z dist init r=3, world=4 2022-11-23T01:20:12.7248096Z dist init r=2, world=4 2022-11-23T01:20:12.7248186Z skip: Need at least 2 CUDA devices (1.819s) 2022-11-23T01:20:12.7248439Z test_transformer_offload_true_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21158 2022-11-23T01:20:12.7248598Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21159 2022-11-23T01:20:12.7248751Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21160 2022-11-23T01:20:12.7248899Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21161 2022-11-23T01:20:12.7249204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7249331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7249611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7249732Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7249907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:20:12.7250174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7250324Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7250604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7250740Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7250917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:20:12.7251182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7251302Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7251568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7251700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7251877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:20:12.7252147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:20:12.7252268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:20:12.7252547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:20:12.7252681Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:20:12.7252854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:20:12.7253140Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7253433Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7253722Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7254014Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:20:12.7254180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:20:12.7254340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:20:12.7254501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:20:12.7254661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:20:12.7254734Z dist init r=2, world=4 2022-11-23T01:20:12.7254794Z dist init r=0, world=4 2022-11-23T01:20:12.7254865Z dist init r=1, world=4 2022-11-23T01:20:12.7254936Z dist init r=3, world=4 2022-11-23T01:20:12.7255040Z skip: Need at least 2 CUDA devices (1.920s) 2022-11-23T01:20:12.7255050Z 2022-11-23T01:20:12.7255246Z ---------------------------------------------------------------------- 2022-11-23T01:20:12.7255323Z Ran 59 tests in 95.938s 2022-11-23T01:20:12.7255328Z 2022-11-23T01:20:12.7255398Z OK (skipped=59) 2022-11-23T01:20:12.7255403Z 2022-11-23T01:20:12.7255549Z Generating XML reports... 2022-11-23T01:20:12.7255847Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestHooks-20221123011836.xml 2022-11-23T01:20:12.7256156Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestNoGrad-20221123011836.xml 2022-11-23T01:20:12.7256463Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParamInit-20221123011836.xml 2022-11-23T01:20:12.7256784Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParityWithDDP-20221123011836.xml 2022-11-23T01:20:12.7256789Z 2022-11-23T01:20:12.7257207Z ##[endgroup] 2022-11-23T01:20:12.7257551Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_core (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_core_0s0_3j9n) 2022-11-23T01:20:12.7257556Z 2022-11-23T01:20:12.7257758Z Running distributed/fsdp/test_fsdp_state_dict ... [2022-11-23 01:20:12.667577] 2022-11-23T01:20:12.7258127Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_state_dict.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:20:12.667900] 2022-11-23T01:23:21.8475977Z 2022-11-23T01:23:21.8476523Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_state_dict 2022-11-23T01:23:21.8492779Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_state_dict (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_state_dict_8bqrw0m9) 2022-11-23T01:23:21.8493442Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict 2022-11-23T01:23:21.8493669Z 2022-11-23T01:23:21.8493747Z Running tests... 2022-11-23T01:23:21.8494042Z ---------------------------------------------------------------------- 2022-11-23T01:23:21.8494506Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8523455Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21225 2022-11-23T01:23:21.8524115Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21226 2022-11-23T01:23:21.8524657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8524991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8525431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8543447Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8545043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8546218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8546948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8547907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8548635Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8549360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8550465Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8551614Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8552531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8553601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8554099Z dist init r=0, world=2 2022-11-23T01:23:21.8554304Z dist init r=1, world=2 2022-11-23T01:23:21.8554512Z skip: Need at least 2 CUDA devices (1.875s) 2022-11-23T01:23:21.8555272Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8556112Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21253 2022-11-23T01:23:21.8556791Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21254 2022-11-23T01:23:21.8557687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8558151Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8558856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8559479Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8560094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8560956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8561343Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8561762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8562113Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8562452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8563160Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8564082Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8564697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8565258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8565672Z dist init r=1, world=2 2022-11-23T01:23:21.8565952Z dist init r=0, world=2 2022-11-23T01:23:21.8566298Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8566990Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8567831Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21281 2022-11-23T01:23:21.8568472Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21282 2022-11-23T01:23:21.8569271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8569871Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8570352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8570733Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8571073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8571711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8572547Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8573497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8574051Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8574603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8575479Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8598266Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8599226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8599993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8600731Z dist init r=1, world=2 2022-11-23T01:23:21.8601118Z dist init r=0, world=2 2022-11-23T01:23:21.8601578Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8602500Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8603654Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21309 2022-11-23T01:23:21.8604490Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21310 2022-11-23T01:23:21.8605510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8606241Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8607193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8607925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8608648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8609690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8610388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8611340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8612094Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8612824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8613905Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8615078Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8615933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8616700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8617242Z dist init r=0, world=2 2022-11-23T01:23:21.8617632Z dist init r=1, world=2 2022-11-23T01:23:21.8618068Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8618969Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8620123Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21337 2022-11-23T01:23:21.8621089Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21338 2022-11-23T01:23:21.8622085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8622732Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8623675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8624420Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8625123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8626224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8626928Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8627857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8628567Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8629285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8630361Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8631492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8632382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8633146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8633675Z dist init r=1, world=2 2022-11-23T01:23:21.8633989Z dist init r=0, world=2 2022-11-23T01:23:21.8634314Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8634738Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8635259Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21365 2022-11-23T01:23:21.8635639Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21366 2022-11-23T01:23:21.8636100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8636433Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8636856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8637204Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8637545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8638021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8638337Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8638770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8639119Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8639458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8639942Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8640469Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8641046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8641386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8641648Z dist init r=1, world=2 2022-11-23T01:23:21.8641833Z dist init r=0, world=2 2022-11-23T01:23:21.8642043Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8642446Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8642967Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21393 2022-11-23T01:23:21.8643401Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21394 2022-11-23T01:23:21.8643866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8644184Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8644617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8644965Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8645294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8645764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8646098Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8646527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8646860Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8647199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8647702Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8648222Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8648597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8649992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8650262Z dist init r=0, world=2 2022-11-23T01:23:21.8650432Z dist init r=1, world=2 2022-11-23T01:23:21.8650641Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8651062Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8651587Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21421 2022-11-23T01:23:21.8651965Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21422 2022-11-23T01:23:21.8652422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8652754Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8653186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8653525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8653867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8654379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8654698Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8655134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8655481Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8655827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8656309Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8656860Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8657249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8657600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8657854Z dist init r=0, world=2 2022-11-23T01:23:21.8658037Z dist init r=1, world=2 2022-11-23T01:23:21.8658247Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8658652Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8659173Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21449 2022-11-23T01:23:21.8659568Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21450 2022-11-23T01:23:21.8660023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8660343Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8660778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8661128Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8661453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8661928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8662259Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8662689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8663024Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8663362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8663860Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8664382Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8664755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8665104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8665367Z dist init r=0, world=2 2022-11-23T01:23:21.8665535Z dist init r=1, world=2 2022-11-23T01:23:21.8665741Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8666160Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8666704Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21477 2022-11-23T01:23:21.8667086Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21478 2022-11-23T01:23:21.8667542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8667877Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8668297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8668645Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8669017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8669494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8669810Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8670241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8670585Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8670921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8671403Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8671925Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8672398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8672737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8673002Z dist init r=1, world=2 2022-11-23T01:23:21.8673188Z dist init r=0, world=2 2022-11-23T01:23:21.8673400Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8673803Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8674326Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21505 2022-11-23T01:23:21.8674723Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21506 2022-11-23T01:23:21.8675185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8675506Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8675936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8676290Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8676615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8677083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8677412Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8677836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8678165Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8678505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8678997Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8679580Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8679954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8680304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8680567Z dist init r=1, world=2 2022-11-23T01:23:21.8680845Z dist init r=0, world=2 2022-11-23T01:23:21.8681058Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8681475Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8682055Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21533 2022-11-23T01:23:21.8682437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21534 2022-11-23T01:23:21.8682897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8683230Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8683644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8683993Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8684330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8684801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8685119Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8685553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8685901Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8686239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8686722Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8687242Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8687627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8687966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8688226Z dist init r=1, world=2 2022-11-23T01:23:21.8688408Z dist init r=0, world=2 2022-11-23T01:23:21.8688619Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8689029Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8689550Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21561 2022-11-23T01:23:21.8689948Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21562 2022-11-23T01:23:21.8690407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8690727Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8691160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8691506Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8691870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8692345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8692674Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8693105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8693434Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8693770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8694304Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8694813Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8695204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8695559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8695820Z dist init r=1, world=2 2022-11-23T01:23:21.8695990Z dist init r=0, world=2 2022-11-23T01:23:21.8696200Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8696621Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8697140Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21589 2022-11-23T01:23:21.8697522Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21590 2022-11-23T01:23:21.8697988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8698324Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8698737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8699080Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8699420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8699894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8700210Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8700642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8700993Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8701318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8701816Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8702337Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8702725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8703065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8703327Z dist init r=0, world=2 2022-11-23T01:23:21.8703511Z dist init r=1, world=2 2022-11-23T01:23:21.8703706Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8704152Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8704677Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21617 2022-11-23T01:23:21.8705069Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21618 2022-11-23T01:23:21.8705512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8705845Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8706275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8706659Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8706986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8707464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8707798Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8708228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8708558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8708892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8709393Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8709906Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8710297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8710648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8710911Z dist init r=1, world=2 2022-11-23T01:23:21.8711079Z dist init r=0, world=2 2022-11-23T01:23:21.8711290Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8711710Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8712297Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21645 2022-11-23T01:23:21.8712696Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21646 2022-11-23T01:23:21.8713161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8713493Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8713911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8714265Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8714604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8715075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8715390Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8715821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8716169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8716489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8717021Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8717543Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8717929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8718263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8718518Z dist init r=1, world=2 2022-11-23T01:23:21.8718702Z dist init r=0, world=2 2022-11-23T01:23:21.8718898Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8719352Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8719881Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21673 2022-11-23T01:23:21.8720272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21674 2022-11-23T01:23:21.8720841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8721177Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8721610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8721957Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8722281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8722753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8723081Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8723516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8723843Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8724184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8724681Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8725189Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8725577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8725928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8726188Z dist init r=1, world=2 2022-11-23T01:23:21.8726358Z dist init r=0, world=2 2022-11-23T01:23:21.8726572Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8726998Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8727506Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21701 2022-11-23T01:23:21.8727894Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21702 2022-11-23T01:23:21.8728352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8728687Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8729108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8729506Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8729845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8730316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8730633Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8731066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8731413Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8731779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8732272Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8732796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8733190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8733524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8733778Z dist init r=0, world=2 2022-11-23T01:23:21.8733962Z dist init r=1, world=2 2022-11-23T01:23:21.8734157Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8734580Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8735105Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21729 2022-11-23T01:23:21.8735497Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21730 2022-11-23T01:23:21.8735940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8736270Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8736700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8737050Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8737376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8737843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8738178Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8738595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8738944Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8739281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8739778Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8740288Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8740676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8741025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8741285Z dist init r=0, world=2 2022-11-23T01:23:21.8741454Z dist init r=1, world=2 2022-11-23T01:23:21.8741662Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8742117Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8742625Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21757 2022-11-23T01:23:21.8743015Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21758 2022-11-23T01:23:21.8743469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8743799Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8744247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8744592Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8744929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8745405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8745720Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8746150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8746493Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8746813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8747313Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8747834Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8748224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8748562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8748815Z dist init r=0, world=2 2022-11-23T01:23:21.8749000Z dist init r=1, world=2 2022-11-23T01:23:21.8749197Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8749618Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8750135Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21785 2022-11-23T01:23:21.8750528Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21786 2022-11-23T01:23:21.8750974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8751302Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8751734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8752086Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8752495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8752967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8753298Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8753719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8754064Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8754440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8754940Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8755448Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8755838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8756184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8756443Z dist init r=0, world=2 2022-11-23T01:23:21.8756646Z dist init r=1, world=2 2022-11-23T01:23:21.8756856Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8757277Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8757778Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21813 2022-11-23T01:23:21.8758166Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21814 2022-11-23T01:23:21.8758619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8758954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8759371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8759724Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8760062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8760535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8760952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8761386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8761736Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8762053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8762548Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8763080Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8763465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8763804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8764062Z dist init r=1, world=2 2022-11-23T01:23:21.8764248Z dist init r=0, world=2 2022-11-23T01:23:21.8764442Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:23:21.8764859Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8765379Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21841 2022-11-23T01:23:21.8765774Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21842 2022-11-23T01:23:21.8766216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8766549Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8767058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8767410Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8767736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8768207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8768542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8768959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8769345Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8769681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8770181Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8770692Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8771082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8771431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8771693Z dist init r=1, world=2 2022-11-23T01:23:21.8771863Z dist init r=0, world=2 2022-11-23T01:23:21.8772071Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8772497Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8773001Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21869 2022-11-23T01:23:21.8773392Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21870 2022-11-23T01:23:21.8773850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8774183Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8774600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8774947Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8775289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8775746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8776076Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8776508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8776851Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8777172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8777667Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8778188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8778581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8778913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8779168Z dist init r=0, world=2 2022-11-23T01:23:21.8779350Z dist init r=1, world=2 2022-11-23T01:23:21.8779582Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:23:21.8780012Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8780530Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21897 2022-11-23T01:23:21.8780923Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21898 2022-11-23T01:23:21.8781368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8781734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8782170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8782517Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8782841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8783316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8783643Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8784061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8784403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8784740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8785236Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8785745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8786135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8786483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8786744Z dist init r=1, world=2 2022-11-23T01:23:21.8786911Z dist init r=0, world=2 2022-11-23T01:23:21.8787120Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8787546Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8788055Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21925 2022-11-23T01:23:21.8788451Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21926 2022-11-23T01:23:21.8788906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8789237Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8789655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8790001Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8790338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8790794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8791129Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8791560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8791936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8792343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8792846Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8793367Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8793753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8794085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8794426Z dist init r=1, world=2 2022-11-23T01:23:21.8794611Z dist init r=0, world=2 2022-11-23T01:23:21.8794805Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8795231Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8795759Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21953 2022-11-23T01:23:21.8796153Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21954 2022-11-23T01:23:21.8796597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8796930Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8797365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8797715Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8798042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8798510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8798838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8799254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8799602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8799937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8800433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8801078Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8801471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8801824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8802076Z dist init r=1, world=2 2022-11-23T01:23:21.8802263Z dist init r=0, world=2 2022-11-23T01:23:21.8802473Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:23:21.8802897Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8803407Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21981 2022-11-23T01:23:21.8803798Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21982 2022-11-23T01:23:21.8804315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8804651Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8805066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8805413Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8805752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8806207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8806535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8807003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8807352Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8807678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8808173Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8808697Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8809084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8809421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8809675Z dist init r=1, world=2 2022-11-23T01:23:21.8809857Z dist init r=0, world=2 2022-11-23T01:23:21.8810053Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:23:21.8810477Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8811000Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22009 2022-11-23T01:23:21.8811390Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22010 2022-11-23T01:23:21.8811830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8812162Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8812591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8812925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8813258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8813733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8814062Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8814476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8814821Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8815158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8815652Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8816165Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8816551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8816932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8817184Z dist init r=0, world=2 2022-11-23T01:23:21.8817368Z dist init r=1, world=2 2022-11-23T01:23:21.8817576Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:23:21.8817994Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8818493Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22037 2022-11-23T01:23:21.8818884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22038 2022-11-23T01:23:21.8819374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8819706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8820125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8820472Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8820806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8821263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8821590Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8822019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8822366Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8822686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8823188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8823712Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8824100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8824435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8824692Z dist init r=1, world=2 2022-11-23T01:23:21.8824874Z dist init r=0, world=2 2022-11-23T01:23:21.8825069Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8825484Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8826007Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22065 2022-11-23T01:23:21.8826397Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22066 2022-11-23T01:23:21.8826840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8827171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8827598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8827930Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8828269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8828744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8829073Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8829522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8829869Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8830206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8830703Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8831211Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8831629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8831980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8832306Z dist init r=1, world=2 2022-11-23T01:23:21.8832488Z dist init r=0, world=2 2022-11-23T01:23:21.8832703Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8833121Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8833632Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22093 2022-11-23T01:23:21.8834025Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22094 2022-11-23T01:23:21.8834484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8834818Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8835236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8835589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8835929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8836382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8836710Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8837139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8837482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8837806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8838305Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8838826Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8839204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8839552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8839808Z dist init r=0, world=2 2022-11-23T01:23:21.8839991Z dist init r=1, world=2 2022-11-23T01:23:21.8840189Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:23:21.8840876Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8841404Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22121 2022-11-23T01:23:21.8841799Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22122 2022-11-23T01:23:21.8842320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8842658Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8843093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8843427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8843766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8844234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8844601Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8845015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8845363Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8845702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8846182Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8846704Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8847094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8847442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8847691Z dist init r=0, world=2 2022-11-23T01:23:21.8847874Z dist init r=1, world=2 2022-11-23T01:23:21.8848081Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8848484Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8848998Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22149 2022-11-23T01:23:21.8849390Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22150 2022-11-23T01:23:21.8849844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8850162Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8850592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8850945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8851286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8851745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8852070Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8852500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8852845Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8853168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8853665Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8854184Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8854610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8854961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8855217Z dist init r=0, world=2 2022-11-23T01:23:21.8855400Z dist init r=1, world=2 2022-11-23T01:23:21.8855595Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8856007Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8856522Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22177 2022-11-23T01:23:21.8856948Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22178 2022-11-23T01:23:21.8857389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8857722Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8858152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8858484Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8858821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8859290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8859617Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8860034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8860381Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8860719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8861199Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8861719Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8862106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8862454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8862701Z dist init r=0, world=2 2022-11-23T01:23:21.8862883Z dist init r=1, world=2 2022-11-23T01:23:21.8863095Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:23:21.8863498Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8864013Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22205 2022-11-23T01:23:21.8864407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22206 2022-11-23T01:23:21.8864860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8865174Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8865603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8865955Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8866293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8866746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8867106Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8867538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8867882Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8868205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8868701Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8869219Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8869621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8869973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8870234Z dist init r=0, world=2 2022-11-23T01:23:21.8870417Z dist init r=1, world=2 2022-11-23T01:23:21.8870613Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8871027Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8871542Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22233 2022-11-23T01:23:21.8871921Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22234 2022-11-23T01:23:21.8872459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8872793Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8873232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8873566Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8873904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8874376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8874706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8875120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8875469Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8875804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8876286Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8876809Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8877201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8877547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8877796Z dist init r=1, world=2 2022-11-23T01:23:21.8877977Z dist init r=0, world=2 2022-11-23T01:23:21.8878185Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8878582Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8879099Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22261 2022-11-23T01:23:21.8879524Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22262 2022-11-23T01:23:21.8879983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8880301Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8880914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8881263Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8881601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8882121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8882450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8882878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8883209Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8883542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8884044Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8884564Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8884937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8885285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8885552Z dist init r=0, world=2 2022-11-23T01:23:21.8885720Z dist init r=1, world=2 2022-11-23T01:23:21.8885929Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8886344Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8886858Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22289 2022-11-23T01:23:21.8887239Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22290 2022-11-23T01:23:21.8887692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8888023Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8888455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8888790Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8889128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8889597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8889927Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8890341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8890691Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8891025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8891509Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8892070Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8892458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8892807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8893056Z dist init r=0, world=2 2022-11-23T01:23:21.8893237Z dist init r=1, world=2 2022-11-23T01:23:21.8893447Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8893841Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8894383Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22317 2022-11-23T01:23:21.8894777Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22318 2022-11-23T01:23:21.8895232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8895550Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8895976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8896325Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8896662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8897117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8897449Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8897878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8898212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8898552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8899048Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8899566Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8899936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8900286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8900551Z dist init r=1, world=2 2022-11-23T01:23:21.8900720Z dist init r=0, world=2 2022-11-23T01:23:21.8900932Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:23:21.8901350Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8901861Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22345 2022-11-23T01:23:21.8902239Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22346 2022-11-23T01:23:21.8902692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8903025Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8903454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8903788Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8904127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8904629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8904947Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8905376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8905721Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8906055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8906535Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8907080Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8907466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8907815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8908064Z dist init r=0, world=2 2022-11-23T01:23:21.8908248Z dist init r=1, world=2 2022-11-23T01:23:21.8908457Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:23:21.8908858Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8909366Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22373 2022-11-23T01:23:21.8909756Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22374 2022-11-23T01:23:21.8910210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8910524Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8910952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8911301Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8911624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8912094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8912499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8912927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8913262Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8913597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8914094Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8914623Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8914996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8915346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8915608Z dist init r=0, world=2 2022-11-23T01:23:21.8915777Z dist init r=1, world=2 2022-11-23T01:23:21.8915985Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8916402Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8916958Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22401 2022-11-23T01:23:21.8917340Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22402 2022-11-23T01:23:21.8917795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8918127Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8918556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8918892Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8919261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8919733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8920047Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8920474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8920999Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8921340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8921823Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8922340Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8922729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8923081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8923330Z dist init r=0, world=2 2022-11-23T01:23:21.8923517Z dist init r=1, world=2 2022-11-23T01:23:21.8923728Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8924122Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8924632Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22429 2022-11-23T01:23:21.8925024Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22430 2022-11-23T01:23:21.8925479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8925797Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8926227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8926578Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8926900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8927373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8927703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8928129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8928462Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8928801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8929298Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8929879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8930258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8930607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8930869Z dist init r=1, world=2 2022-11-23T01:23:21.8931039Z dist init r=0, world=2 2022-11-23T01:23:21.8931248Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8931661Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8932330Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22457 2022-11-23T01:23:21.8932708Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22458 2022-11-23T01:23:21.8933164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8933495Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8933911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8934259Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8934596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8935172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8935485Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8935914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8936260Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8936597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8937079Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8937600Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8937992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8938332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8938596Z dist init r=0, world=2 2022-11-23T01:23:21.8938778Z dist init r=1, world=2 2022-11-23T01:23:21.8938984Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8939385Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8939892Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22485 2022-11-23T01:23:21.8940281Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22486 2022-11-23T01:23:21.8940996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8941504Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8942160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8942622Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8943202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8943862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8944367Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8945050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8945604Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8946156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8947123Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8948017Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8948696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8949306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8949706Z dist init r=1, world=2 2022-11-23T01:23:21.8949987Z dist init r=0, world=2 2022-11-23T01:23:21.8950302Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8950799Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8951315Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22513 2022-11-23T01:23:21.8951715Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22514 2022-11-23T01:23:21.8952280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8952602Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8953034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8953381Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8953718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8954181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8954509Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8954940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8955271Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8955605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8956099Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8956620Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8957158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8957683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8957944Z dist init r=0, world=2 2022-11-23T01:23:21.8958119Z dist init r=1, world=2 2022-11-23T01:23:21.8958326Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8958733Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8959331Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22541 2022-11-23T01:23:21.8959712Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22542 2022-11-23T01:23:21.8960171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8960503Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8961130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8961535Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8961868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8962345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8962661Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8963088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8963436Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8963769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8964250Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8964775Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8965160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8965505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8965752Z dist init r=1, world=2 2022-11-23T01:23:21.8965936Z dist init r=0, world=2 2022-11-23T01:23:21.8966143Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8966513Z test_fsdp_state_dict_keys_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22569 2022-11-23T01:23:21.8966932Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22570 2022-11-23T01:23:21.8967383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8967717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8968130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8968473Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8968814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8969269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8969594Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8970020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8970361Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8970684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8971176Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8971695Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8972126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8972461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8972717Z dist init r=1, world=2 2022-11-23T01:23:21.8972895Z dist init r=0, world=2 2022-11-23T01:23:21.8973088Z skip: Need at least 2 CUDA devices (1.613s) 2022-11-23T01:23:21.8973470Z test_fsdp_state_dict_keys_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22597 2022-11-23T01:23:21.8973886Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22598 2022-11-23T01:23:21.8974370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8974689Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8975116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8975463Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8975788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8976254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8976581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8977006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8977336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8977667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8978160Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8978674Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8979045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8979392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8979649Z dist init r=1, world=2 2022-11-23T01:23:21.8979817Z dist init r=0, world=2 2022-11-23T01:23:21.8980023Z skip: Need at least 2 CUDA devices (1.613s) 2022-11-23T01:23:21.8980398Z test_fsdp_state_dict_keys_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22625 2022-11-23T01:23:21.8980798Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22626 2022-11-23T01:23:21.8981250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8981581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8982005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8982339Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8982673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8983135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8983467Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8983878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8984222Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8984586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8985067Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8985585Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8985970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8986317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8986595Z dist init r=0, world=2 2022-11-23T01:23:21.8986774Z dist init r=1, world=2 2022-11-23T01:23:21.8986980Z skip: Need at least 2 CUDA devices (1.613s) 2022-11-23T01:23:21.8987352Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8987959Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22653 2022-11-23T01:23:21.8988359Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22654 2022-11-23T01:23:21.8988635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8988758Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8989040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8989164Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8989342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8989616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8989739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8990015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8990147Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8990321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8990623Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8990920Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8991072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8991235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8991309Z dist init r=1, world=2 2022-11-23T01:23:21.8991380Z dist init r=0, world=2 2022-11-23T01:23:21.8991482Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8991734Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8992075Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22681 2022-11-23T01:23:21.8992316Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22682 2022-11-23T01:23:21.8992595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8992707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8993060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8993196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8993372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8993639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8993761Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8994035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8994209Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8994370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8994667Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8994964Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8995128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8995288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8995363Z dist init r=1, world=2 2022-11-23T01:23:21.8995436Z dist init r=0, world=2 2022-11-23T01:23:21.8995538Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.8995770Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8996111Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22709 2022-11-23T01:23:21.8996268Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22710 2022-11-23T01:23:21.8996541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8996663Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8996943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8997076Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8997253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.8997532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.8997642Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.8997919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.8998051Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.8998225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.8998524Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8998819Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.8998983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.8999147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.8999220Z dist init r=1, world=2 2022-11-23T01:23:21.8999280Z dist init r=0, world=2 2022-11-23T01:23:21.8999382Z skip: Need at least 2 CUDA devices (1.613s) 2022-11-23T01:23:21.8999656Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.8999997Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22737 2022-11-23T01:23:21.9000151Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22738 2022-11-23T01:23:21.9000424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9000547Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9001033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9001155Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9001333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9001603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9001725Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9002004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9002138Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9002312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9002612Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9002909Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9003062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9003222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9003296Z dist init r=0, world=2 2022-11-23T01:23:21.9003369Z dist init r=1, world=2 2022-11-23T01:23:21.9003474Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9003717Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9004053Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22765 2022-11-23T01:23:21.9004209Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22766 2022-11-23T01:23:21.9004479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9004592Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9004869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9005004Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9005179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9005447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9005568Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9005845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9005978Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9006196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9006495Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9006787Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9006953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9007114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9007190Z dist init r=0, world=2 2022-11-23T01:23:21.9007302Z dist init r=1, world=2 2022-11-23T01:23:21.9007403Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9007629Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9007966Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22793 2022-11-23T01:23:21.9008122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22794 2022-11-23T01:23:21.9008392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9008513Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9008792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9008925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9009102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9009373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9009483Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9009762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9009896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9010069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9010367Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9010661Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9010827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9010988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9011067Z dist init r=0, world=2 2022-11-23T01:23:21.9011128Z dist init r=1, world=2 2022-11-23T01:23:21.9011231Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:23:21.9011490Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9011827Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22821 2022-11-23T01:23:21.9011983Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22822 2022-11-23T01:23:21.9012254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9012376Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9012702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9012837Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9013001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9013270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9013391Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9013667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9013798Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9014001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9014299Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9014596Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9014746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9014908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9014982Z dist init r=1, world=2 2022-11-23T01:23:21.9015055Z dist init r=0, world=2 2022-11-23T01:23:21.9015157Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9015414Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9015756Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22849 2022-11-23T01:23:21.9015913Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22850 2022-11-23T01:23:21.9016184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9016294Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9016572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9016705Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9016881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9017151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9017271Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9017549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9017681Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9017843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9018138Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9018433Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9018596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9018759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9018834Z dist init r=1, world=2 2022-11-23T01:23:21.9018906Z dist init r=0, world=2 2022-11-23T01:23:21.9019009Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9019270Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9019611Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22877 2022-11-23T01:23:21.9019765Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22878 2022-11-23T01:23:21.9020035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9020156Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9020465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9020598Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9020774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9021040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9021148Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9021427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9021557Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9021731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9022024Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9022320Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9022484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9022644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9022722Z dist init r=0, world=2 2022-11-23T01:23:21.9022781Z dist init r=1, world=2 2022-11-23T01:23:21.9022883Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9023129Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9023463Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22905 2022-11-23T01:23:21.9023621Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22906 2022-11-23T01:23:21.9023894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9024017Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9024299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9024431Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9024594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9024863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9024985Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9025263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9025396Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9025572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9025893Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9026192Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9026342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9026504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9026576Z dist init r=1, world=2 2022-11-23T01:23:21.9026648Z dist init r=0, world=2 2022-11-23T01:23:21.9026785Z skip: Need at least 2 CUDA devices (1.613s) 2022-11-23T01:23:21.9027033Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9027372Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22933 2022-11-23T01:23:21.9027526Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22934 2022-11-23T01:23:21.9027796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9027906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9028185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9028319Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9028497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9028764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9028887Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9029163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9029298Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9029460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9029754Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9030048Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9030212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9030374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9030448Z dist init r=1, world=2 2022-11-23T01:23:21.9030521Z dist init r=0, world=2 2022-11-23T01:23:21.9030623Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9030867Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9031191Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22961 2022-11-23T01:23:21.9031349Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22962 2022-11-23T01:23:21.9031620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9031745Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9032021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9032274Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9032451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9032724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9032833Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9033110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9033241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9033444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9033740Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9034038Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9034202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9034363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9034441Z dist init r=1, world=2 2022-11-23T01:23:21.9034502Z dist init r=0, world=2 2022-11-23T01:23:21.9034604Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9034840Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9035180Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22989 2022-11-23T01:23:21.9035335Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22990 2022-11-23T01:23:21.9035609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9035732Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9036011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9036145Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9036308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9036580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9036706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9036985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9037120Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9037296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9037589Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9037885Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9038036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9038195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9038271Z dist init r=1, world=2 2022-11-23T01:23:21.9038343Z dist init r=0, world=2 2022-11-23T01:23:21.9038445Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9038709Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9039050Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23017 2022-11-23T01:23:21.9039208Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23018 2022-11-23T01:23:21.9039480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9039590Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9039872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9040034Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9040209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9040483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9040734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9041084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9041216Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9041379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9041678Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9041975Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9042136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9042299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9042373Z dist init r=1, world=2 2022-11-23T01:23:21.9042445Z dist init r=0, world=2 2022-11-23T01:23:21.9042546Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9042770Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9043107Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23045 2022-11-23T01:23:21.9043262Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23046 2022-11-23T01:23:21.9043536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9043658Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9043940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9044073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9044249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9044517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9044624Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9044901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9045035Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9045211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9045568Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9045864Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9046028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9046189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9046262Z dist init r=0, world=2 2022-11-23T01:23:21.9046321Z dist init r=1, world=2 2022-11-23T01:23:21.9046425Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9046659Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9047038Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23073 2022-11-23T01:23:21.9047195Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23074 2022-11-23T01:23:21.9047468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9047589Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9047868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9048001Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9048166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9048438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9048560Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9048839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9048972Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9049146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9049440Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9049737Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9049888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9050052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9050126Z dist init r=0, world=2 2022-11-23T01:23:21.9050197Z dist init r=1, world=2 2022-11-23T01:23:21.9050300Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9050550Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9050886Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23101 2022-11-23T01:23:21.9051041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23102 2022-11-23T01:23:21.9051314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9051424Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9051705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9051839Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9052044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9052316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9052436Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9052715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9052845Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9053010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9053333Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9053627Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9053795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9053955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9054030Z dist init r=1, world=2 2022-11-23T01:23:21.9054106Z dist init r=0, world=2 2022-11-23T01:23:21.9054210Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9054444Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9054783Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23129 2022-11-23T01:23:21.9054940Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23130 2022-11-23T01:23:21.9055212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9055336Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9055617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9055750Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9055926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9056193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9056302Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9056579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9056713Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9056887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9057185Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9057480Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9057643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9057804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9057878Z dist init r=0, world=2 2022-11-23T01:23:21.9057941Z dist init r=1, world=2 2022-11-23T01:23:21.9058043Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9058280Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9058664Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23157 2022-11-23T01:23:21.9058823Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23158 2022-11-23T01:23:21.9059099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9059221Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9059499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9059632Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9059822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9060091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9060213Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9060487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9060620Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9060794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9061091Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9061385Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9061539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9061701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9061774Z dist init r=0, world=2 2022-11-23T01:23:21.9061850Z dist init r=1, world=2 2022-11-23T01:23:21.9061954Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9062189Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9062527Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23185 2022-11-23T01:23:21.9062682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23186 2022-11-23T01:23:21.9062954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9063066Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9063349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9063485Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9063660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9063929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9064051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9064331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9064464Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9064631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9064931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9065257Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9065424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9065587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9065661Z dist init r=0, world=2 2022-11-23T01:23:21.9065732Z dist init r=1, world=2 2022-11-23T01:23:21.9065835Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9066065Z test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9066328Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23213 2022-11-23T01:23:21.9066483Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23214 2022-11-23T01:23:21.9066761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9066883Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9067162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9067297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9067476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9067744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9067854Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9068130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9068264Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9068438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9068737Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9069033Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9069197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9069359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9069438Z dist init r=1, world=2 2022-11-23T01:23:21.9069497Z dist init r=0, world=2 2022-11-23T01:23:21.9069599Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9069839Z test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9070072Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23241 2022-11-23T01:23:21.9070226Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23242 2022-11-23T01:23:21.9070496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9070617Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9070892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9071027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9071190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9071488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9071611Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9071885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9072017Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9072262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9072561Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9072892Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9073043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9073207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9073281Z dist init r=0, world=2 2022-11-23T01:23:21.9073355Z dist init r=1, world=2 2022-11-23T01:23:21.9073457Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:23:21.9073696Z test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9073930Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23269 2022-11-23T01:23:21.9074087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23270 2022-11-23T01:23:21.9074359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9074468Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9074749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9074884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9075060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9075330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9075450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9075726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9075859Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9076020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9076316Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9076613Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9076776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9076937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9077010Z dist init r=1, world=2 2022-11-23T01:23:21.9077083Z dist init r=0, world=2 2022-11-23T01:23:21.9077185Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:23:21.9077412Z test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9077675Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23297 2022-11-23T01:23:21.9077834Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23298 2022-11-23T01:23:21.9078108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9078234Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9078511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9078643Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9078819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9079118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9079227Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9079508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9079642Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9079816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9080109Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9080404Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9080685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9080905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9080980Z dist init r=1, world=2 2022-11-23T01:23:21.9081039Z dist init r=0, world=2 2022-11-23T01:23:21.9081145Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9081391Z test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9081621Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23325 2022-11-23T01:23:21.9081775Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23326 2022-11-23T01:23:21.9082058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9082181Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9082461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9082595Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9082761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9083027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9083147Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9083423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9083555Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9083728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9084028Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9084390Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9084544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9084704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9084777Z dist init r=0, world=2 2022-11-23T01:23:21.9084851Z dist init r=1, world=2 2022-11-23T01:23:21.9084954Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:23:21.9085198Z test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9085432Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23353 2022-11-23T01:23:21.9085622Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23354 2022-11-23T01:23:21.9085895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9086005Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9086285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9086421Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9086597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9086864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9086987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9087262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9087395Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9087557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9087852Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9088146Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9088307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9088468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9088544Z dist init r=1, world=2 2022-11-23T01:23:21.9088618Z dist init r=0, world=2 2022-11-23T01:23:21.9088720Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9088952Z test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9089186Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23381 2022-11-23T01:23:21.9089341Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23382 2022-11-23T01:23:21.9089615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9089735Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9090013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9090148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9090324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9090626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9090736Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9091017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9091151Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9091326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9091621Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9091983Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9092144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9092308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9092382Z dist init r=1, world=2 2022-11-23T01:23:21.9092442Z dist init r=0, world=2 2022-11-23T01:23:21.9092544Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9092786Z test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9093019Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23409 2022-11-23T01:23:21.9093174Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23410 2022-11-23T01:23:21.9093448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9093570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9093851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9093986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9094149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9094417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9094539Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9094814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9094948Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9095122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9095419Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9095711Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9095863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9096024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9096097Z dist init r=1, world=2 2022-11-23T01:23:21.9096170Z dist init r=0, world=2 2022-11-23T01:23:21.9096273Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9096510Z test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9096746Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23437 2022-11-23T01:23:21.9096928Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23438 2022-11-23T01:23:21.9097201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9097311Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9097590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9097723Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9097897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9098198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9098319Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9098598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9098731Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9098895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9099192Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9099487Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9099652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9099819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9099894Z dist init r=0, world=2 2022-11-23T01:23:21.9099966Z dist init r=1, world=2 2022-11-23T01:23:21.9100069Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9100292Z test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9100527Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23465 2022-11-23T01:23:21.9100680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23466 2022-11-23T01:23:21.9100951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9101073Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9101355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9101489Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9101669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9101939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9102050Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9102329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9102462Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9102637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9102935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9103232Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9103438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9103601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9103674Z dist init r=1, world=2 2022-11-23T01:23:21.9103734Z dist init r=0, world=2 2022-11-23T01:23:21.9103837Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9104074Z test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9104307Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23493 2022-11-23T01:23:21.9104492Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23494 2022-11-23T01:23:21.9104766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9104890Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9105169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9105306Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9105469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9105737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9105859Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9106140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9106271Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9106446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9106740Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9107036Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9107188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9107349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9107423Z dist init r=0, world=2 2022-11-23T01:23:21.9107496Z dist init r=1, world=2 2022-11-23T01:23:21.9107600Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9107832Z test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9108066Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23521 2022-11-23T01:23:21.9108221Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23522 2022-11-23T01:23:21.9108491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9108603Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9108883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9109016Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9109193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9109461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9109610Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9109889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9110022Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9110185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9129602Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9130020Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9130311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9130465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9130534Z dist init r=1, world=2 2022-11-23T01:23:21.9130609Z dist init r=0, world=2 2022-11-23T01:23:21.9130710Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9130935Z test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9131262Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23549 2022-11-23T01:23:21.9131418Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23550 2022-11-23T01:23:21.9131687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9131801Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9132076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9132212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9132388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9132654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9132769Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9133043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9133171Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9133345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9133632Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9133926Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9134086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9134243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9134310Z dist init r=1, world=2 2022-11-23T01:23:21.9134376Z dist init r=0, world=2 2022-11-23T01:23:21.9134470Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9134695Z test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9135010Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23577 2022-11-23T01:23:21.9135158Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23578 2022-11-23T01:23:21.9135473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9135600Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9135882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9136016Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9136192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9136466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9136621Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9136889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9137025Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9137204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9137505Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9137803Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9137970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9138136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9138213Z dist init r=0, world=2 2022-11-23T01:23:21.9138275Z dist init r=1, world=2 2022-11-23T01:23:21.9138378Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9138609Z test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9138938Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23605 2022-11-23T01:23:21.9139093Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23606 2022-11-23T01:23:21.9139365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9139487Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9139763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9139900Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9140063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9140334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9140454Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9140730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9140864Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9141040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9141337Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9141634Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9141796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9141976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9142052Z dist init r=0, world=2 2022-11-23T01:23:21.9142125Z dist init r=1, world=2 2022-11-23T01:23:21.9142227Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9142454Z test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9142780Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23633 2022-11-23T01:23:21.9142934Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23634 2022-11-23T01:23:21.9143241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9143352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9143633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9143766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9143944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9144216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9144336Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9144617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9144753Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9144916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9145214Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9145508Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9145670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9145832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9145905Z dist init r=0, world=2 2022-11-23T01:23:21.9145976Z dist init r=1, world=2 2022-11-23T01:23:21.9146078Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9146302Z test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9146611Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23661 2022-11-23T01:23:21.9146767Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23662 2022-11-23T01:23:21.9147037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9147158Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9147435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9147567Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9147740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9148009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9148133Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9148427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9148562Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9148737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9149034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9149333Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9149495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9149687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9149761Z dist init r=0, world=2 2022-11-23T01:23:21.9149820Z dist init r=1, world=2 2022-11-23T01:23:21.9149923Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9150147Z test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9150467Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23689 2022-11-23T01:23:21.9150620Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23690 2022-11-23T01:23:21.9150891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9151012Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9151290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9151426Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9151589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9151861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9151983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9152353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9152490Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9152667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9152969Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9153267Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9153418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9153583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9153660Z dist init r=0, world=2 2022-11-23T01:23:21.9153735Z dist init r=1, world=2 2022-11-23T01:23:21.9153841Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9154063Z test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9154383Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23717 2022-11-23T01:23:21.9154544Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23718 2022-11-23T01:23:21.9154817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9154925Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9155236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9155373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9155546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9155816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9155937Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9156214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9156377Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9156541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9156838Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9157131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9157294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9157456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9157530Z dist init r=0, world=2 2022-11-23T01:23:21.9157601Z dist init r=1, world=2 2022-11-23T01:23:21.9157704Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9157913Z test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9158233Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23745 2022-11-23T01:23:21.9158393Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23746 2022-11-23T01:23:21.9158666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9158789Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9159067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9159201Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9159376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9159648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9159757Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9160037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9160169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9160341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9160779Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9161145Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9161310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9161477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9161551Z dist init r=1, world=2 2022-11-23T01:23:21.9161610Z dist init r=0, world=2 2022-11-23T01:23:21.9161784Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9161959Z test_state_dict_rank0_offload_save_load_flow_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9162181Z Tests saving a model checkpoint only on rank 0 and loading it only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23773 2022-11-23T01:23:21.9162337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23774 2022-11-23T01:23:21.9162613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9162738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9163057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9163178Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9163356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9163625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9163750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9164033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9164170Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9164340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9164627Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9164918Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9165071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9165225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9165300Z dist init r=1, world=2 2022-11-23T01:23:21.9165370Z dist init r=0, world=2 2022-11-23T01:23:21.9165474Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9165646Z test_state_dict_rank0_offload_save_load_flow_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9165872Z Tests saving a model checkpoint only on rank 0 and loading it only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23801 2022-11-23T01:23:21.9166025Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23802 2022-11-23T01:23:21.9166287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9166408Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9166688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9166823Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9166997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9167266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9167386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9167667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9167802Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9167965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9168291Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9168590Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9168753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9168911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9168986Z dist init r=0, world=2 2022-11-23T01:23:21.9169057Z dist init r=1, world=2 2022-11-23T01:23:21.9169161Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9169434Z test_state_dict_save_load_flow_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23829 2022-11-23T01:23:21.9169588Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23830 2022-11-23T01:23:21.9169862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9169982Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9170263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9170396Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9170574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9170842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9170953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9171231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9171369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9171545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9171839Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9172134Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9172296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9172457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9172533Z dist init r=1, world=2 2022-11-23T01:23:21.9172592Z dist init r=0, world=2 2022-11-23T01:23:21.9172694Z skip: Need at least 2 CUDA devices (1.613s) 2022-11-23T01:23:21.9172949Z test_state_dict_save_load_flow_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23857 2022-11-23T01:23:21.9173107Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23858 2022-11-23T01:23:21.9173375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9173499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9173779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9173912Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9174077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9174346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9174468Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9174778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9174913Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9175086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9175383Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9175680Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9175873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9176022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9176095Z dist init r=1, world=2 2022-11-23T01:23:21.9176170Z dist init r=0, world=2 2022-11-23T01:23:21.9176271Z skip: Need at least 2 CUDA devices (1.613s) 2022-11-23T01:23:21.9176519Z test_state_dict_save_load_flow_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23885 2022-11-23T01:23:21.9176674Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23886 2022-11-23T01:23:21.9176944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9177068Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9177335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9177470Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9177649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9177922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9178044Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9178321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9178453Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9178632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9178928Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9179211Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9179373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9179538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9179613Z dist init r=0, world=2 2022-11-23T01:23:21.9179685Z dist init r=1, world=2 2022-11-23T01:23:21.9179787Z skip: Need at least 2 CUDA devices (1.613s) 2022-11-23T01:23:21.9180059Z test_state_dict_skip_module_state_dict_type_local_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23913 2022-11-23T01:23:21.9180213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23914 2022-11-23T01:23:21.9180475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9180599Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9180877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9181040Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9181216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9181486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9181606Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9181885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9182018Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9182210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9182509Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9182804Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9182965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9183130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9183203Z dist init r=0, world=2 2022-11-23T01:23:21.9183274Z dist init r=1, world=2 2022-11-23T01:23:21.9183374Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:23:21.9183634Z test_state_dict_skip_module_state_dict_type_sharded_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23941 2022-11-23T01:23:21.9183795Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23942 2022-11-23T01:23:21.9184065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9184187Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9184465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9184596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9184771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9185041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9185161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9185427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9185561Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9185738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9186033Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9186327Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9186488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9186650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9186723Z dist init r=0, world=2 2022-11-23T01:23:21.9186785Z dist init r=1, world=2 2022-11-23T01:23:21.9186887Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:23:21.9187154Z test_state_dict_skip_module_state_dict_type_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23969 2022-11-23T01:23:21.9187336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23970 2022-11-23T01:23:21.9187608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9187732Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9188013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9188147Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9188321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9188607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9188730Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9189012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9189145Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9189323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9189617Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9189915Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9190076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9190240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9190301Z dist init r=0, world=2 2022-11-23T01:23:21.9190372Z dist init r=1, world=2 2022-11-23T01:23:21.9190478Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:23:21.9190695Z test_state_dict_type (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23997 2022-11-23T01:23:21.9190848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23998 2022-11-23T01:23:21.9191116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9191238Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9191513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9191635Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9191814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9192082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9192280Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9192565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9192702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9192881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9193178Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9193474Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9193626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9193791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9193904Z dist init r=0, world=2 2022-11-23T01:23:21.9193980Z dist init r=1, world=2 2022-11-23T01:23:21.9194085Z skip: Need at least 2 CUDA devices (1.613s) 2022-11-23T01:23:21.9194386Z test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24025 2022-11-23T01:23:21.9194542Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24026 2022-11-23T01:23:21.9194814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9194957Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9195241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9195377Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9195558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9195828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9195953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9196237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9196374Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9196538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9196836Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9197131Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9197292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9197456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9197528Z dist init r=1, world=2 2022-11-23T01:23:21.9197601Z dist init r=0, world=2 2022-11-23T01:23:21.9197703Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9198002Z test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24053 2022-11-23T01:23:21.9198147Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24054 2022-11-23T01:23:21.9198418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9198539Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9198824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9198958Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9199135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9199405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9199527Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9199802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9199927Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9200101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9200431Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9200938Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9201127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9201290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9201365Z dist init r=1, world=2 2022-11-23T01:23:21.9201439Z dist init r=0, world=2 2022-11-23T01:23:21.9201528Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:23:21.9201881Z test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24081 2022-11-23T01:23:21.9202041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24082 2022-11-23T01:23:21.9202323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9202447Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9202734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9202870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9203048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9203326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9203436Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9203721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9203855Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9204034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9204332Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9204631Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9204793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9204960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9205021Z dist init r=0, world=2 2022-11-23T01:23:21.9205092Z dist init r=1, world=2 2022-11-23T01:23:21.9205197Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9205492Z test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24109 2022-11-23T01:23:21.9205645Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24110 2022-11-23T01:23:21.9205917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9206039Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9206319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9206454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9206617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9206971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9207101Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9207379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9207514Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9207690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9207988Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9208326Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9208489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9208643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9208719Z dist init r=1, world=2 2022-11-23T01:23:21.9208794Z dist init r=0, world=2 2022-11-23T01:23:21.9208896Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:23:21.9209184Z test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24137 2022-11-23T01:23:21.9209338Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24138 2022-11-23T01:23:21.9209609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9209733Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9210001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9210137Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9210313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9210584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9210707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9210993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9211131Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9211309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9211608Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9211897Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9212061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9212221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9212294Z dist init r=1, world=2 2022-11-23T01:23:21.9212368Z dist init r=0, world=2 2022-11-23T01:23:21.9212470Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9212758Z test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24165 2022-11-23T01:23:21.9212914Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24166 2022-11-23T01:23:21.9213174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9213327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9213609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9213744Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9213921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9214191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9214314Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9214591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9214764Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9214926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9215232Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9215527Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9215690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9215850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9215924Z dist init r=0, world=2 2022-11-23T01:23:21.9215995Z dist init r=1, world=2 2022-11-23T01:23:21.9216096Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9216371Z test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24193 2022-11-23T01:23:21.9216528Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24194 2022-11-23T01:23:21.9216802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9216925Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9217204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9217337Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9217512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9217788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9217907Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9218176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9218308Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9218484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9218779Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9219075Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9219237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9219401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9219476Z dist init r=0, world=2 2022-11-23T01:23:21.9219537Z dist init r=1, world=2 2022-11-23T01:23:21.9219640Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:23:21.9219954Z test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24221 2022-11-23T01:23:21.9220110Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24222 2022-11-23T01:23:21.9220381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9220502Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9220780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9220942Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9221118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9221377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9221499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9221775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9221908Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9222082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9222376Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9222674Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9222835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9222997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9223058Z dist init r=1, world=2 2022-11-23T01:23:21.9223129Z dist init r=0, world=2 2022-11-23T01:23:21.9223231Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:23:21.9223441Z test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9223673Z Tests saving and loading a state dict for a model manually wrapped with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24249 2022-11-23T01:23:21.9223828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24250 2022-11-23T01:23:21.9224101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9224224Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9224494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9224627Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9224803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9225071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9225194Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9225473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9225608Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9225786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9226074Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9226395Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9226558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9226720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9226796Z dist init r=0, world=2 2022-11-23T01:23:21.9226869Z dist init r=1, world=2 2022-11-23T01:23:21.9226970Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9227181Z test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9227439Z Tests saving and loading a state dict for a model manually wrapped with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24277 2022-11-23T01:23:21.9227581Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24278 2022-11-23T01:23:21.9227858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9227980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9228259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9228393Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9228568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9228839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9228962Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9229230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9229368Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9229545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9229839Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9230133Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9230294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9230457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9230535Z dist init r=0, world=2 2022-11-23T01:23:21.9230606Z dist init r=1, world=2 2022-11-23T01:23:21.9230696Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9230894Z test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9231126Z Tests saving and loading a state dict for a model manually wrapped with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24305 2022-11-23T01:23:21.9231282Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24306 2022-11-23T01:23:21.9231553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9231674Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9231950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9232086Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9232329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9232642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9232765Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9233045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9233180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9233357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9233656Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9233982Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9234143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9234293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9234368Z dist init r=0, world=2 2022-11-23T01:23:21.9234438Z dist init r=1, world=2 2022-11-23T01:23:21.9234541Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:23:21.9234741Z test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T01:23:21.9234972Z Tests saving and loading a state dict for a model manually wrapped with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24333 2022-11-23T01:23:21.9235129Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24334 2022-11-23T01:23:21.9235404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9235514Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9235797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9235932Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9236109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9236379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9236504Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9236782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9236917Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9237093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9237377Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9237672Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9237834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9237997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9238072Z dist init r=1, world=2 2022-11-23T01:23:21.9238143Z dist init r=0, world=2 2022-11-23T01:23:21.9238247Z skip: Need at least 2 CUDA devices (1.615s) 2022-11-23T01:23:21.9238515Z test_state_dict_with_shared_parameters_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24361 2022-11-23T01:23:21.9238661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24362 2022-11-23T01:23:21.9238965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9239090Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9239368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9239501Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9239675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9239946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9240098Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9240378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9240498Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9240859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9241252Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9241549Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9241713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9241876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9241954Z dist init r=1, world=2 2022-11-23T01:23:21.9242028Z dist init r=0, world=2 2022-11-23T01:23:21.9242120Z skip: Need at least 2 CUDA devices (1.614s) 2022-11-23T01:23:21.9242392Z test_state_dict_with_shared_parameters_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24389 2022-11-23T01:23:21.9242547Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24390 2022-11-23T01:23:21.9242824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9242947Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9243227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9243364Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9243541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9243813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9243922Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9244207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9244341Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9244518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9244816Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9245114Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9245282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9245448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9245522Z dist init r=0, world=2 2022-11-23T01:23:21.9245583Z dist init r=1, world=2 2022-11-23T01:23:21.9245744Z skip: Need at least 2 CUDA devices (1.616s) 2022-11-23T01:23:21.9246008Z test_state_dict_with_shared_parameters_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24417 2022-11-23T01:23:21.9246167Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24418 2022-11-23T01:23:21.9246441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9246566Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9246851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9247025Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9247191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9247465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9247591Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9247873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9248010Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9248189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9248488Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9248789Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9248953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9249105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9249178Z dist init r=1, world=2 2022-11-23T01:23:21.9249250Z dist init r=0, world=2 2022-11-23T01:23:21.9249354Z skip: Need at least 2 CUDA devices (1.613s) 2022-11-23T01:23:21.9249577Z test_wrong_state_dict_config (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24445 2022-11-23T01:23:21.9249730Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24446 2022-11-23T01:23:21.9250002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9250128Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9250396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9250532Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9250706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:23:21.9250975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:23:21.9251097Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:23:21.9251377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:23:21.9251511Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:23:21.9251684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:23:21.9251973Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9252315Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:23:21.9252481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:23:21.9252643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:23:21.9252719Z dist init r=1, world=2 2022-11-23T01:23:21.9252793Z dist init r=0, world=2 2022-11-23T01:23:21.9252896Z skip: Need at least 2 CUDA devices (1.613s) 2022-11-23T01:23:21.9252908Z 2022-11-23T01:23:21.9253108Z ---------------------------------------------------------------------- 2022-11-23T01:23:21.9253175Z Ran 116 tests in 187.518s 2022-11-23T01:23:21.9253223Z 2022-11-23T01:23:21.9253284Z OK (skipped=116) 2022-11-23T01:23:21.9253289Z 2022-11-23T01:23:21.9253373Z Generating XML reports... 2022-11-23T01:23:21.9253717Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict/TEST-TestFSDPStateDict-20221123012013.xml 2022-11-23T01:23:21.9253722Z 2022-11-23T01:23:21.9254185Z ##[endgroup] 2022-11-23T01:23:21.9254553Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_state_dict (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_state_dict_8bqrw0m9) 2022-11-23T01:23:21.9254558Z 2022-11-23T01:23:21.9254752Z Running distributed/rpc/test_faulty_agent ... [2022-11-23 01:23:21.851175] 2022-11-23T01:23:21.9255144Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/test_faulty_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:23:21.851529] 2022-11-23T01:27:54.3099060Z 2022-11-23T01:27:54.3099632Z Expand the folded group to see the log file of distributed/rpc/test_faulty_agent 2022-11-23T01:27:54.3108215Z ##[group]PRINTING LOG FILE of distributed/rpc/test_faulty_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_faulty_agent_0h0w8bg_) 2022-11-23T01:27:54.3108855Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprl3vw_84 2022-11-23T01:27:54.3109344Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprl3vw_84/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3110171Z , <__main__.FaultyFaultyAgentDistAutogradTest testMethod=test_verify_backend_options>]> 2022-11-23T01:27:54.3161987Z test_context_cleanup_tensor_with_grad (__main__.FaultyFaultyAgentDistAutogradTest) 2022-11-23T01:27:54.3162540Z test_verify_backend_options (__main__.FaultyFaultyAgentDistAutogradTest) 2022-11-23T01:27:54.3165906Z , <__main__.FaultyFaultyAgentRpcTest testMethod=test_builtin_remote_message_dropped_timeout_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_check_failed_messages>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_custom_faulty_messages>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_custom_messages_to_delay>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_no_faulty_messages>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_builtin_delay_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_builtin_delay_timeout_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_dropped_pickle>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_dropped_pickle_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_script_delay_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_script_delay_timeout_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_rpc_builtin_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_rpc_script_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_rref_to_here_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_udf_remote_message_delay_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_udf_remote_message_delay_timeout_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_udf_remote_message_dropped_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_udf_remote_message_dropped_timeout_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_verify_backend_options>]> 2022-11-23T01:27:54.3169067Z test_builtin_remote_message_dropped_timeout (__main__.FaultyFaultyAgentRpcTest) 2022-11-23T01:27:54.3169548Z test_builtin_remote_message_dropped_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) 2022-11-23T01:27:54.3170027Z test_check_failed_messages (__main__.FaultyFaultyAgentRpcTest) 2022-11-23T01:27:54.3170445Z test_custom_faulty_messages (__main__.FaultyFaultyAgentRpcTest) 2022-11-23T01:27:54.3170991Z test_custom_messages_to_delay (__main__.FaultyFaultyAgentRpcTest) 2022-11-23T01:27:54.3171379Z test_no_faulty_messages (__main__.FaultyFaultyAgentRpcTest) 2022-11-23T01:27:54.3171794Z test_remote_message_builtin_delay_timeout (__main__.FaultyFaultyAgentRpcTest) 2022-11-23T01:27:54.3172260Z test_remote_message_builtin_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) 2022-11-23T01:27:54.3172742Z test_remote_message_dropped_pickle (__main__.FaultyFaultyAgentRpcTest) 2022-11-23T01:27:54.3173198Z test_remote_message_dropped_pickle_to_self (__main__.FaultyFaultyAgentRpcTest) 2022-11-23T01:27:54.3173670Z test_remote_message_script_delay_timeout (__main__.FaultyFaultyAgentRpcTest) 2022-11-23T01:27:54.3174124Z test_remote_message_script_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) 2022-11-23T01:27:54.3174538Z test_rpc_builtin_timeout (__main__.FaultyFaultyAgentRpcTest) 2022-11-23T01:27:54.3174945Z test_rpc_script_timeout (__main__.FaultyFaultyAgentRpcTest) 2022-11-23T01:27:54.3175379Z test_rref_to_here_timeout (__main__.FaultyFaultyAgentRpcTest) 2022-11-23T01:27:54.3200298Z test_udf_remote_message_delay_timeout (__main__.FaultyFaultyAgentRpcTest) 2022-11-23T01:27:54.3200991Z test_udf_remote_message_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) 2022-11-23T01:27:54.3201654Z test_udf_remote_message_dropped_timeout (__main__.FaultyFaultyAgentRpcTest) 2022-11-23T01:27:54.3202061Z test_udf_remote_message_dropped_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) 2022-11-23T01:27:54.3202388Z test_verify_backend_options (__main__.FaultyFaultyAgentRpcTest) 2022-11-23T01:27:54.3203732Z , <__main__.FaultyJitFaultyAgentRpcTest testMethod=test_rref_timeout_pickle_in_jit>, <__main__.FaultyJitFaultyAgentRpcTest testMethod=test_rref_timeout_pickle_script_func>, <__main__.FaultyJitFaultyAgentRpcTest testMethod=test_rref_to_here_timeout_in_jit>, <__main__.FaultyJitFaultyAgentRpcTest testMethod=test_timeout_in_python>, <__main__.FaultyJitFaultyAgentRpcTest testMethod=test_timeout_in_torchscript_function>]> 2022-11-23T01:27:54.3204621Z test_remote_timeout_to_here_in_jit (__main__.FaultyJitFaultyAgentRpcTest) 2022-11-23T01:27:54.3205096Z test_rref_timeout_pickle_in_jit (__main__.FaultyJitFaultyAgentRpcTest) 2022-11-23T01:27:54.3205423Z test_rref_timeout_pickle_script_func (__main__.FaultyJitFaultyAgentRpcTest) 2022-11-23T01:27:54.3205749Z test_rref_to_here_timeout_in_jit (__main__.FaultyJitFaultyAgentRpcTest) 2022-11-23T01:27:54.3206044Z test_timeout_in_python (__main__.FaultyJitFaultyAgentRpcTest) 2022-11-23T01:27:54.3206578Z test_timeout_in_torchscript_function (__main__.FaultyJitFaultyAgentRpcTest) 2022-11-23T01:27:54.3207526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3208059Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3208741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3209278Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3209826Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg5ff_550 2022-11-23T01:27:54.3210699Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg5ff_550/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3211186Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-11-23T01:27:54.3211488Z 2022-11-23T01:27:54.3211642Z Running tests... 2022-11-23T01:27:54.3212204Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3212755Z test_context_cleanup_tensor_with_grad (__main__.FaultyFaultyAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24494 2022-11-23T01:27:54.3213266Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24495 2022-11-23T01:27:54.3213789Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24496 2022-11-23T01:27:54.3214363Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24497 2022-11-23T01:27:54.3215113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3215792Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3216430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3216780Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3217125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5hfuado7 2022-11-23T01:27:54.3217515Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5hfuado7/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3218009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3218336Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3218877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3219439Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3220071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpefnnwpv5 2022-11-23T01:27:54.3220754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpefnnwpv5/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3221617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3222038Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3222469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3222820Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3223150Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5g9sqxmr 2022-11-23T01:27:54.3223583Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5g9sqxmr/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3224454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3224786Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3225507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3226056Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3226670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpowa_pmgv 2022-11-23T01:27:54.3227388Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpowa_pmgv/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3228083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:27:54.3228819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:27:54.3229469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:27:54.3230014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:27:54.3230373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:27:54.3230742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:27:54.3231103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:27:54.3231492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:27:54.3231988Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:27:54.3232606Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:27:54.3233131Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:27:54.3233632Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:27:54.3233926Z ok (9.692s) 2022-11-23T01:27:54.3234031Z 2022-11-23T01:27:54.3234233Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3234464Z Ran 1 test in 9.692s 2022-11-23T01:27:54.3234579Z 2022-11-23T01:27:54.3234644Z OK 2022-11-23T01:27:54.3234736Z 2022-11-23T01:27:54.3234821Z Generating XML reports... 2022-11-23T01:27:54.3235322Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentDistAutogradTest-20221123012324.xml 2022-11-23T01:27:54.3235884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3236216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3236642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3236974Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3237315Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0rw32ara 2022-11-23T01:27:54.3237714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0rw32ara/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3238172Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-11-23T01:27:54.3238371Z 2022-11-23T01:27:54.3238431Z Running tests... 2022-11-23T01:27:54.3238730Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3239155Z test_verify_backend_options (__main__.FaultyFaultyAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24681 2022-11-23T01:27:54.3239579Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24682 2022-11-23T01:27:54.3239894Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24683 2022-11-23T01:27:54.3240215Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24684 2022-11-23T01:27:54.3240816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3241196Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3241915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3242543Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3243269Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjal947vx 2022-11-23T01:27:54.3243978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjal947vx/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3244890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3245498Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3246186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3246535Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3246934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk7qxoo9b 2022-11-23T01:27:54.3247334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk7qxoo9b/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3247812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3248139Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3248776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3249351Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3249876Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpevsy02d8 2022-11-23T01:27:54.3250605Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpevsy02d8/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3251669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3252253Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3253039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3253393Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3253739Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx9_682mo 2022-11-23T01:27:54.3254123Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx9_682mo/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3254496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:27:54.3254845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:27:54.3255176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:27:54.3255520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:27:54.3255768Z ok (8.289s) 2022-11-23T01:27:54.3255870Z 2022-11-23T01:27:54.3256070Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3256300Z Ran 1 test in 8.289s 2022-11-23T01:27:54.3256413Z 2022-11-23T01:27:54.3256476Z OK 2022-11-23T01:27:54.3256567Z 2022-11-23T01:27:54.3256650Z Generating XML reports... 2022-11-23T01:27:54.3257145Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentDistAutogradTest-20221123012336.xml 2022-11-23T01:27:54.3257718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3258049Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3258478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3258816Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3259161Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9_62zssw 2022-11-23T01:27:54.3259619Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9_62zssw/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3260076Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-11-23T01:27:54.3260261Z 2022-11-23T01:27:54.3260336Z Running tests... 2022-11-23T01:27:54.3260641Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3261067Z test_builtin_remote_message_dropped_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24856 2022-11-23T01:27:54.3261476Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24857 2022-11-23T01:27:54.3261840Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24858 2022-11-23T01:27:54.3262168Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24859 2022-11-23T01:27:54.3262620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3262941Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3263370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3263716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3264048Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppy5p1xgc 2022-11-23T01:27:54.3264454Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppy5p1xgc/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3264939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3265269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3265693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3266022Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3266363Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpydt1wy4z 2022-11-23T01:27:54.3266761Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpydt1wy4z/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3267231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3267559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3267991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3268334Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3268659Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpntavvikg 2022-11-23T01:27:54.3269061Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpntavvikg/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3269546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3269873Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3270282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3270622Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3270963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi43u4x96 2022-11-23T01:27:54.3271346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi43u4x96/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3271720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:27:54.3272071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:27:54.3272541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:27:54.3272872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:27:54.3273117Z ok (8.391s) 2022-11-23T01:27:54.3273220Z 2022-11-23T01:27:54.3273419Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3273646Z Ran 1 test in 8.391s 2022-11-23T01:27:54.3273756Z 2022-11-23T01:27:54.3273817Z OK 2022-11-23T01:27:54.3273906Z 2022-11-23T01:27:54.3273990Z Generating XML reports... 2022-11-23T01:27:54.3274464Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012346.xml 2022-11-23T01:27:54.3275121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3275447Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3275863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3276204Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3276546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbapuna7w 2022-11-23T01:27:54.3276937Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbapuna7w/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3277399Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-11-23T01:27:54.3277599Z 2022-11-23T01:27:54.3277671Z Running tests... 2022-11-23T01:27:54.3277971Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3278385Z test_builtin_remote_message_dropped_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25031 2022-11-23T01:27:54.3278809Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25032 2022-11-23T01:27:54.3279137Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25033 2022-11-23T01:27:54.3279460Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25034 2022-11-23T01:27:54.3279900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3280230Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3280817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3281155Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3281501Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnhap067a 2022-11-23T01:27:54.3281899Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnhap067a/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3282569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3283023Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3283734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3284270Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3284795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgro7g1j4 2022-11-23T01:27:54.3285440Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgro7g1j4/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3286255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3286789Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3287614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3288172Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3288722Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7r5ixwpm 2022-11-23T01:27:54.3289352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7r5ixwpm/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3290114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3290639Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3291443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3292073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3292689Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpax5f4i1i 2022-11-23T01:27:54.3293421Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpax5f4i1i/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3294105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:27:54.3294727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:27:54.3295345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:27:54.3295972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:27:54.3296315Z ok (8.389s) 2022-11-23T01:27:54.3296409Z 2022-11-23T01:27:54.3296611Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3296853Z Ran 1 test in 8.389s 2022-11-23T01:27:54.3296964Z 2022-11-23T01:27:54.3297024Z OK 2022-11-23T01:27:54.3297116Z 2022-11-23T01:27:54.3297187Z Generating XML reports... 2022-11-23T01:27:54.3297660Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012356.xml 2022-11-23T01:27:54.3298206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3298532Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3298946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3299288Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3299633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprydxnvhi 2022-11-23T01:27:54.3300026Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprydxnvhi/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3300486Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-11-23T01:27:54.3300688Z 2022-11-23T01:27:54.3300761Z Running tests... 2022-11-23T01:27:54.3301063Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3301450Z test_check_failed_messages (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25206 2022-11-23T01:27:54.3301854Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25207 2022-11-23T01:27:54.3302179Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25208 2022-11-23T01:27:54.3302501Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25209 2022-11-23T01:27:54.3302939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3303266Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3303738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3304072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3304416Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxpyux6sy 2022-11-23T01:27:54.3304823Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxpyux6sy/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3305309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3305622Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3306046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3306420Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3306747Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjmc35oo2 2022-11-23T01:27:54.3307145Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjmc35oo2/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3307626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3307953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3308365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3308706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3309048Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1zuq0bz_ 2022-11-23T01:27:54.3309448Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1zuq0bz_/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3309924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3310249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3310673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3311007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3311346Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp49i61hf6 2022-11-23T01:27:54.3311741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp49i61hf6/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3312187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:27:54.3312527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:27:54.3312865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:27:54.3313209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:27:54.3313447Z ok (12.895s) 2022-11-23T01:27:54.3313549Z 2022-11-23T01:27:54.3313751Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3313992Z Ran 1 test in 12.895s 2022-11-23T01:27:54.3314103Z 2022-11-23T01:27:54.3314164Z OK 2022-11-23T01:27:54.3314242Z 2022-11-23T01:27:54.3314324Z Generating XML reports... 2022-11-23T01:27:54.3314794Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012406.xml 2022-11-23T01:27:54.3315340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3315659Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3316083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3316432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3316813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn2zrv6oc 2022-11-23T01:27:54.3317202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn2zrv6oc/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3317654Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-11-23T01:27:54.3317855Z 2022-11-23T01:27:54.3317926Z Running tests... 2022-11-23T01:27:54.3318215Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3318615Z test_custom_faulty_messages (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25381 2022-11-23T01:27:54.3319050Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25382 2022-11-23T01:27:54.3319379Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25383 2022-11-23T01:27:54.3319696Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25384 2022-11-23T01:27:54.3320146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3320475Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3321068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3321403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3321747Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp48up5swy 2022-11-23T01:27:54.3322145Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp48up5swy/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3322621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3322946Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3323373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3323713Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3324043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv6vrayls 2022-11-23T01:27:54.3324440Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv6vrayls/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3324919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3325245Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3325660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3326004Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3326348Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppq1ga6vn 2022-11-23T01:27:54.3326741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppq1ga6vn/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3327227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3327552Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3327978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3328306Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3328647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptzexovip 2022-11-23T01:27:54.3329048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptzexovip/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3329472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:27:54.3329824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:27:54.3330162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:27:54.3330504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:27:54.3330741Z ok (8.391s) 2022-11-23T01:27:54.3330844Z 2022-11-23T01:27:54.3331044Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3331282Z Ran 1 test in 8.391s 2022-11-23T01:27:54.3331394Z 2022-11-23T01:27:54.3331443Z OK 2022-11-23T01:27:54.3331573Z 2022-11-23T01:27:54.3331656Z Generating XML reports... 2022-11-23T01:27:54.3332126Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012421.xml 2022-11-23T01:27:54.3332675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3332992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3333417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3333763Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3334094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc_on9ve3 2022-11-23T01:27:54.3334489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc_on9ve3/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3334949Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-11-23T01:27:54.3335153Z 2022-11-23T01:27:54.3335224Z Running tests... 2022-11-23T01:27:54.3335514Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3335923Z test_custom_messages_to_delay (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25556 2022-11-23T01:27:54.3336326Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25557 2022-11-23T01:27:54.3336655Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25558 2022-11-23T01:27:54.3336965Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25559 2022-11-23T01:27:54.3337411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3337739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3338155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3338499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3338843Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3rhf3fz1 2022-11-23T01:27:54.3339244Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3rhf3fz1/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3339714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3340041Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3340465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3340811Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3341139Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc3zp9tla 2022-11-23T01:27:54.3341545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc3zp9tla/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3342030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3342379Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3342812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3343155Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3343497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx667kp1m 2022-11-23T01:27:54.3343883Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx667kp1m/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3344366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3344724Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3345137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3345483Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3346319Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprwu0b9a8 2022-11-23T01:27:54.3346714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprwu0b9a8/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3347076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:27:54.3347423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:27:54.3347766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:27:54.3348110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:27:54.3348347Z ok (2.386s) 2022-11-23T01:27:54.3348449Z 2022-11-23T01:27:54.3348649Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3348887Z Ran 1 test in 2.387s 2022-11-23T01:27:54.3348999Z 2022-11-23T01:27:54.3349049Z OK 2022-11-23T01:27:54.3349138Z 2022-11-23T01:27:54.3349220Z Generating XML reports... 2022-11-23T01:27:54.3349690Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012431.xml 2022-11-23T01:27:54.3350234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3350547Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3350977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3351322Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3351655Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsrv6cgis 2022-11-23T01:27:54.3352053Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsrv6cgis/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3352750Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-11-23T01:27:54.3352951Z 2022-11-23T01:27:54.3353022Z Running tests... 2022-11-23T01:27:54.3353314Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3353713Z test_no_faulty_messages (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25731 2022-11-23T01:27:54.3354110Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25732 2022-11-23T01:27:54.3354427Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25733 2022-11-23T01:27:54.3354753Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25734 2022-11-23T01:27:54.3355201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3355577Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3355992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3356335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3356679Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbjc2vz32 2022-11-23T01:27:54.3357080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbjc2vz32/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3357550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3357916Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3358338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3358666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3359009Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps5h89fv_ 2022-11-23T01:27:54.3359410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps5h89fv_/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3359893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3360206Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3360751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3361101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3361433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp1vt4y35 2022-11-23T01:27:54.3361833Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp1vt4y35/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3362328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3362655Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3363072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3363414Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3363755Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvecq6qy4 2022-11-23T01:27:54.3364153Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvecq6qy4/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3364514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:27:54.3364862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:27:54.3365204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:27:54.3365534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:27:54.3365783Z ok (8.294s) 2022-11-23T01:27:54.3365887Z 2022-11-23T01:27:54.3366084Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3366323Z Ran 1 test in 8.295s 2022-11-23T01:27:54.3366422Z 2022-11-23T01:27:54.3366482Z OK 2022-11-23T01:27:54.3366571Z 2022-11-23T01:27:54.3366655Z Generating XML reports... 2022-11-23T01:27:54.3367125Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012436.xml 2022-11-23T01:27:54.3367658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3367984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3368473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3368821Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3369151Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9aqn_0j6 2022-11-23T01:27:54.3369550Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9aqn_0j6/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3370006Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-11-23T01:27:54.3370205Z 2022-11-23T01:27:54.3370268Z Running tests... 2022-11-23T01:27:54.3370567Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3371030Z test_remote_message_builtin_delay_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25906 2022-11-23T01:27:54.3371444Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25907 2022-11-23T01:27:54.3371762Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25908 2022-11-23T01:27:54.3372086Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25909 2022-11-23T01:27:54.3372542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3372872Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3373284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3373629Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3373974Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7wd9btce 2022-11-23T01:27:54.3374362Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7wd9btce/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3374850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3375173Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3375596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3375927Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3376263Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq7yv_ff2 2022-11-23T01:27:54.3376659Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq7yv_ff2/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3377127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3377455Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3377881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3378221Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3378549Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcp00oz79 2022-11-23T01:27:54.3378945Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcp00oz79/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3379425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3379751Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3380160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3380503Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3380844Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxhg2lt4w 2022-11-23T01:27:54.3381279Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxhg2lt4w/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3381660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:27:54.3382005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:27:54.3382349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:27:54.3382679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:27:54.3382924Z ok (7.386s) 2022-11-23T01:27:54.3383026Z 2022-11-23T01:27:54.3383224Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3383479Z Ran 1 test in 7.386s 2022-11-23T01:27:54.3383590Z 2022-11-23T01:27:54.3383651Z OK 2022-11-23T01:27:54.3383739Z 2022-11-23T01:27:54.3383821Z Generating XML reports... 2022-11-23T01:27:54.3384294Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012446.xml 2022-11-23T01:27:54.3384831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3385160Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3385590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3385924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3386264Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpukzlb4aj 2022-11-23T01:27:54.3386666Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpukzlb4aj/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3387120Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-11-23T01:27:54.3387318Z 2022-11-23T01:27:54.3387377Z Running tests... 2022-11-23T01:27:54.3387678Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3388097Z test_remote_message_builtin_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26081 2022-11-23T01:27:54.3388520Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26082 2022-11-23T01:27:54.3388835Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26083 2022-11-23T01:27:54.3389154Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26084 2022-11-23T01:27:54.3389604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3389924Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3390354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3390700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3391044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppeu9mduq 2022-11-23T01:27:54.3391436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppeu9mduq/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3391921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3392325Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3392742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3393087Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3393429Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiul7scit 2022-11-23T01:27:54.3393876Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiul7scit/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3394354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3394679Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3395107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3395450Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3395778Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq1x91vwb 2022-11-23T01:27:54.3396176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq1x91vwb/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3396695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3397006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3397434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3397776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3398117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkg3zm8u3 2022-11-23T01:27:54.3398501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkg3zm8u3/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3398876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:27:54.3399222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:27:54.3399555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:27:54.3399897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:27:54.3400463Z [E request_callback_no_python.cpp:559] Received error while processing request type 260: false INTERNAL ASSERT FAILED at "/var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp":395, please report a bug to PyTorch. Expected OwnerRRef with id GloballyUniqueId(created_on=0, local_id=0) to be created. 2022-11-23T01:27:54.3401242Z Exception raised from getOwnerRRef at /var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp:395 (most recent call first): 2022-11-23T01:27:54.3401922Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x6b (0x7ffb7dba387b in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-11-23T01:27:54.3402674Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string, std::allocator > const&) + 0xce (0x7ffb7db9f0de in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-11-23T01:27:54.3403476Z frame #2: c10::detail::torchInternalAssertFail(char const*, char const*, unsigned int, char const*, std::__cxx11::basic_string, std::allocator > const&) + 0x4e (0x7ffb7dba0fee in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-11-23T01:27:54.3404227Z frame #3: torch::distributed::rpc::RRefContext::getOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, bool) + 0x440 (0x7ffb825f5790 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-11-23T01:27:54.3405212Z frame #4: torch::distributed::rpc::RequestCallbackNoPython::assignOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, torch::distributed::rpc::GloballyUniqueId const&, c10::intrusive_ptr >) const + 0x5bc (0x7ffb825e581c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-11-23T01:27:54.3406254Z frame #5: torch::distributed::rpc::RequestCallbackImpl::processScriptRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector >) const + 0x12a (0x7ffb8a999b3a in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-11-23T01:27:54.3407197Z frame #6: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x14c (0x7ffb825ea05c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-11-23T01:27:54.3408174Z frame #7: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x65 (0x7ffb8a994c35 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-11-23T01:27:54.3408894Z frame #8: + 0x49f84a3 (0x7ffb825e34a3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-11-23T01:27:54.3409596Z frame #9: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector >) const + 0x538 (0x7ffb825e4038 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-11-23T01:27:54.3410416Z frame #10: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector >) const + 0x57 (0x7ffb825de347 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-11-23T01:27:54.3411008Z frame #11: + 0x4a28012 (0x7ffb82613012 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-11-23T01:27:54.3411523Z frame #12: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7ffb7db91beb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-11-23T01:27:54.3411898Z frame #13: + 0xdbbf4 (0x7ffb7da13bf4 in /opt/conda/lib/libstdc++.so.6) 2022-11-23T01:27:54.3412295Z frame #14: + 0x8609 (0x7ffb92849609 in /lib/x86_64-linux-gnu/libpthread.so.0) 2022-11-23T01:27:54.3412662Z frame #15: clone + 0x43 (0x7ffb9276e133 in /lib/x86_64-linux-gnu/libc.so.6) 2022-11-23T01:27:54.3412820Z 2022-11-23T01:27:54.3412888Z ok (4.383s) 2022-11-23T01:27:54.3412989Z 2022-11-23T01:27:54.3413174Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3413416Z Ran 1 test in 4.383s 2022-11-23T01:27:54.3413529Z 2022-11-23T01:27:54.3413588Z OK 2022-11-23T01:27:54.3413677Z 2022-11-23T01:27:54.3413761Z Generating XML reports... 2022-11-23T01:27:54.3414219Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012455.xml 2022-11-23T01:27:54.3414773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3415107Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3415521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3415866Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3416209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9tln224h 2022-11-23T01:27:54.3416611Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9tln224h/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3417057Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-11-23T01:27:54.3417255Z 2022-11-23T01:27:54.3417329Z Running tests... 2022-11-23T01:27:54.3417628Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3418030Z test_remote_message_dropped_pickle (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26256 2022-11-23T01:27:54.3418472Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26257 2022-11-23T01:27:54.3418803Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26258 2022-11-23T01:27:54.3419133Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26259 2022-11-23T01:27:54.3419573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3419899Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3420332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3420708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3421040Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp69pt4om9 2022-11-23T01:27:54.3421522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3421848Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3422258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3422605Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3422949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1omcelhw 2022-11-23T01:27:54.3423349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp69pt4om9/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3423742Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1omcelhw/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3424232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3424559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3424974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3425319Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3425662Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwbqm8_4z 2022-11-23T01:27:54.3426056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwbqm8_4z/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3426526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3426851Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3427277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3427619Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3427950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3uvs9p0b 2022-11-23T01:27:54.3428348Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3uvs9p0b/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3428725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:27:54.3429059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:27:54.3429402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:27:54.3429738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:27:54.3429990Z ok (8.298s) 2022-11-23T01:27:54.3430079Z 2022-11-23T01:27:54.3430276Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3430514Z Ran 1 test in 8.298s 2022-11-23T01:27:54.3430625Z 2022-11-23T01:27:54.3430685Z OK 2022-11-23T01:27:54.3430775Z 2022-11-23T01:27:54.3430882Z Generating XML reports... 2022-11-23T01:27:54.3431351Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012501.xml 2022-11-23T01:27:54.3431897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3432314Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3432728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3433073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3433454Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvg0tl72a 2022-11-23T01:27:54.3433841Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvg0tl72a/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3434300Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-11-23T01:27:54.3434502Z 2022-11-23T01:27:54.3434576Z Running tests... 2022-11-23T01:27:54.3434872Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3435279Z test_remote_message_dropped_pickle_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26431 2022-11-23T01:27:54.3435752Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26432 2022-11-23T01:27:54.3436160Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26433 2022-11-23T01:27:54.3436505Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26434 2022-11-23T01:27:54.3437108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3437501Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3457742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3458119Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3458482Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjs2pgpm1 2022-11-23T01:27:54.3458879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjs2pgpm1/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3459370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3459695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3460136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3460468Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3460818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvo8fo2k1 2022-11-23T01:27:54.3461220Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvo8fo2k1/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3461710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3462029Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3462459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3462804Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3463137Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6f91zyb5 2022-11-23T01:27:54.3463540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6f91zyb5/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3464027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3464486Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3464908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3465259Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3465609Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplkkx1qxq 2022-11-23T01:27:54.3466004Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplkkx1qxq/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3466385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:27:54.3466790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:27:54.3467135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:27:54.3467471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:27:54.3467725Z ok (8.293s) 2022-11-23T01:27:54.3467829Z 2022-11-23T01:27:54.3468036Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3468266Z Ran 1 test in 8.293s 2022-11-23T01:27:54.3468378Z 2022-11-23T01:27:54.3468441Z OK 2022-11-23T01:27:54.3468532Z 2022-11-23T01:27:54.3468620Z Generating XML reports... 2022-11-23T01:27:54.3469096Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012511.xml 2022-11-23T01:27:54.3469633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3469965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3470397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3470744Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3471081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt4hqiebe 2022-11-23T01:27:54.3471485Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt4hqiebe/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3471943Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-11-23T01:27:54.3472231Z 2022-11-23T01:27:54.3472292Z Running tests... 2022-11-23T01:27:54.3472598Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3473017Z test_remote_message_script_delay_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26606 2022-11-23T01:27:54.3473438Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26607 2022-11-23T01:27:54.3473758Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26608 2022-11-23T01:27:54.3474088Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26609 2022-11-23T01:27:54.3474543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3474864Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3475295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3475645Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3475989Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeep9baky 2022-11-23T01:27:54.3476383Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeep9baky/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3476869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3477235Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3477667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3478000Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3478345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgosc8knj 2022-11-23T01:27:54.3478747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgosc8knj/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3479222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3479582Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3480009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3480356Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3480861Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx_h6c9re 2022-11-23T01:27:54.3481254Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx_h6c9re/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3481731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3482038Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3482455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3482794Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3483133Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpac4ex_dq 2022-11-23T01:27:54.3483523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpac4ex_dq/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3483902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:27:54.3484243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:27:54.3484588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:27:54.3484918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:27:54.3485162Z ok (7.388s) 2022-11-23T01:27:54.3485263Z 2022-11-23T01:27:54.3485459Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3485686Z Ran 1 test in 7.388s 2022-11-23T01:27:54.3485798Z 2022-11-23T01:27:54.3485857Z OK 2022-11-23T01:27:54.3485946Z 2022-11-23T01:27:54.3486027Z Generating XML reports... 2022-11-23T01:27:54.3486488Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012522.xml 2022-11-23T01:27:54.3487035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3487363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3487795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3488126Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3488468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiaogrerh 2022-11-23T01:27:54.3488869Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiaogrerh/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3489333Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-11-23T01:27:54.3489521Z 2022-11-23T01:27:54.3489593Z Running tests... 2022-11-23T01:27:54.3489891Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3490372Z test_remote_message_script_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26781 2022-11-23T01:27:54.3490785Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26782 2022-11-23T01:27:54.3491117Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26783 2022-11-23T01:27:54.3491446Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26784 2022-11-23T01:27:54.3491897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3492262Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3492695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3493042Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3493385Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp28x5vnlg 2022-11-23T01:27:54.3493769Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp28x5vnlg/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3494252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3494578Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3494990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3495332Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3495681Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc77fa5fh 2022-11-23T01:27:54.3496084Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc77fa5fh/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3496556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3496881Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3497307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3497636Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3497975Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4u699wwe 2022-11-23T01:27:54.3498360Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4u699wwe/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3498830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3499136Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3499549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3499881Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3500222Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpabdxvffm 2022-11-23T01:27:54.3500610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpabdxvffm/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3500989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:27:54.3501332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:27:54.3501665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:27:54.3502013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:27:54.3502609Z [E request_callback_no_python.cpp:559] Received error while processing request type 260: false INTERNAL ASSERT FAILED at "/var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp":395, please report a bug to PyTorch. Expected OwnerRRef with id GloballyUniqueId(created_on=0, local_id=0) to be created. 2022-11-23T01:27:54.3503216Z Exception raised from getOwnerRRef at /var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp:395 (most recent call first): 2022-11-23T01:27:54.3503883Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x6b (0x7f4550de887b in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-11-23T01:27:54.3504616Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string, std::allocator > const&) + 0xce (0x7f4550de40de in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-11-23T01:27:54.3505464Z frame #2: c10::detail::torchInternalAssertFail(char const*, char const*, unsigned int, char const*, std::__cxx11::basic_string, std::allocator > const&) + 0x4e (0x7f4550de5fee in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-11-23T01:27:54.3506210Z frame #3: torch::distributed::rpc::RRefContext::getOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, bool) + 0x440 (0x7f455583a790 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-11-23T01:27:54.3507187Z frame #4: torch::distributed::rpc::RequestCallbackNoPython::assignOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, torch::distributed::rpc::GloballyUniqueId const&, c10::intrusive_ptr >) const + 0x5bc (0x7f455582a81c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-11-23T01:27:54.3508180Z frame #5: torch::distributed::rpc::RequestCallbackImpl::processScriptRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector >) const + 0x12a (0x7f455dbdeb3a in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-11-23T01:27:54.3509115Z frame #6: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x14c (0x7f455582f05c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-11-23T01:27:54.3510091Z frame #7: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x65 (0x7f455dbd9c35 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-11-23T01:27:54.3510749Z frame #8: + 0x49f84a3 (0x7f45558284a3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-11-23T01:27:54.3511454Z frame #9: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector >) const + 0x538 (0x7f4555829038 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-11-23T01:27:54.3512350Z frame #10: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector >) const + 0x57 (0x7f4555823347 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-11-23T01:27:54.3512948Z frame #11: + 0x4a28012 (0x7f4555858012 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-11-23T01:27:54.3513462Z frame #12: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f4550dd6beb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-11-23T01:27:54.3513819Z frame #13: + 0xdbbf4 (0x7f4550c58bf4 in /opt/conda/lib/libstdc++.so.6) 2022-11-23T01:27:54.3514264Z frame #14: + 0x8609 (0x7f4565a8e609 in /lib/x86_64-linux-gnu/libpthread.so.0) 2022-11-23T01:27:54.3514636Z frame #15: clone + 0x43 (0x7f45659b3133 in /lib/x86_64-linux-gnu/libc.so.6) 2022-11-23T01:27:54.3514794Z 2022-11-23T01:27:54.3514862Z ok (4.283s) 2022-11-23T01:27:54.3514953Z 2022-11-23T01:27:54.3515151Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3515394Z Ran 1 test in 4.283s 2022-11-23T01:27:54.3515508Z 2022-11-23T01:27:54.3515571Z OK 2022-11-23T01:27:54.3515664Z 2022-11-23T01:27:54.3515736Z Generating XML reports... 2022-11-23T01:27:54.3516213Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012531.xml 2022-11-23T01:27:54.3516800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3517134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3517553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3517898Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3518245Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp608mdxc5 2022-11-23T01:27:54.3518634Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp608mdxc5/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3519093Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-11-23T01:27:54.3519294Z 2022-11-23T01:27:54.3519367Z Running tests... 2022-11-23T01:27:54.3519673Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3520061Z test_rpc_builtin_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26956 2022-11-23T01:27:54.3520462Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26957 2022-11-23T01:27:54.3520946Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26958 2022-11-23T01:27:54.3521271Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26959 2022-11-23T01:27:54.3521733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3522068Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3522501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3522838Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3523185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7x1edoji 2022-11-23T01:27:54.3523589Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7x1edoji/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3524079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3524395Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3524823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3525169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3525498Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_0mcb6ni 2022-11-23T01:27:54.3525898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_0mcb6ni/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3526386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3526715Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3527199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3527549Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3527897Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpol3nd007 2022-11-23T01:27:54.3528285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpol3nd007/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3528774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3529106Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3529537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3529911Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3530254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa87gzd39 2022-11-23T01:27:54.3530656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa87gzd39/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3531033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:27:54.3531369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:27:54.3531708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:27:54.3532053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:27:54.3532291Z ok (15.801s) 2022-11-23T01:27:54.3532392Z 2022-11-23T01:27:54.3532596Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3532839Z Ran 1 test in 15.801s 2022-11-23T01:27:54.3532952Z 2022-11-23T01:27:54.3533012Z OK 2022-11-23T01:27:54.3533090Z 2022-11-23T01:27:54.3533172Z Generating XML reports... 2022-11-23T01:27:54.3533644Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012537.xml 2022-11-23T01:27:54.3534194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3534511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3534936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3535277Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3535623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppvlnh2oo 2022-11-23T01:27:54.3536013Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppvlnh2oo/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3536469Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-11-23T01:27:54.3536669Z 2022-11-23T01:27:54.3536741Z Running tests... 2022-11-23T01:27:54.3537029Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3537425Z test_rpc_script_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27131 2022-11-23T01:27:54.3537824Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27132 2022-11-23T01:27:54.3538155Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27133 2022-11-23T01:27:54.3538472Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27134 2022-11-23T01:27:54.3538920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3539250Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3539668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3540055Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3540399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4t1jp7ms 2022-11-23T01:27:54.3540797Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4t1jp7ms/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3541270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3541596Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3542019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3542412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3542743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfn6op9by 2022-11-23T01:27:54.3543147Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfn6op9by/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3543634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3543948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3544374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3544713Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3545054Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp34a0w4og 2022-11-23T01:27:54.3545440Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp34a0w4og/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3545931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3546261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3546691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3547023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3547362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2bqrrcrx 2022-11-23T01:27:54.3547762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2bqrrcrx/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3548125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:27:54.3548473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:27:54.3548820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:27:54.3549163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:27:54.3549402Z ok (9.891s) 2022-11-23T01:27:54.3549503Z 2022-11-23T01:27:54.3549704Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3549940Z Ran 1 test in 9.891s 2022-11-23T01:27:54.3550051Z 2022-11-23T01:27:54.3550099Z OK 2022-11-23T01:27:54.3550188Z 2022-11-23T01:27:54.3550272Z Generating XML reports... 2022-11-23T01:27:54.3550744Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012555.xml 2022-11-23T01:27:54.3551291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3551605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3552040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3552470Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3552847Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfvxla_in 2022-11-23T01:27:54.3553251Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfvxla_in/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3553712Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-11-23T01:27:54.3553911Z 2022-11-23T01:27:54.3553983Z Running tests... 2022-11-23T01:27:54.3554274Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3554673Z test_rref_to_here_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27306 2022-11-23T01:27:54.3555115Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27307 2022-11-23T01:27:54.3555432Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27308 2022-11-23T01:27:54.3555760Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27309 2022-11-23T01:27:54.3556213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3556541Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3556954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3557298Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3557638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfngikquu 2022-11-23T01:27:54.3558041Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfngikquu/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3558517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3558846Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3559271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3559602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3559942Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxt25uo8f 2022-11-23T01:27:54.3560341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxt25uo8f/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3561013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3561331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3561766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3562113Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3562445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp34j2eqer 2022-11-23T01:27:54.3562848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp34j2eqer/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3563331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3563660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3564071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3564416Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3564758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphiqick59 2022-11-23T01:27:54.3565157Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphiqick59/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3565518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:27:54.3565930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:27:54.3566280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:27:54.3566614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:27:54.3566872Z ok (4.382s) 2022-11-23T01:27:54.3566975Z 2022-11-23T01:27:54.3567178Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3567420Z Ran 1 test in 4.382s 2022-11-23T01:27:54.3567519Z 2022-11-23T01:27:54.3567581Z OK 2022-11-23T01:27:54.3567674Z 2022-11-23T01:27:54.3567761Z Generating XML reports... 2022-11-23T01:27:54.3568276Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012606.xml 2022-11-23T01:27:54.3568813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3569146Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3569576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3569923Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3570254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcjnefsvt 2022-11-23T01:27:54.3570662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcjnefsvt/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3571124Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-11-23T01:27:54.3571330Z 2022-11-23T01:27:54.3571390Z Running tests... 2022-11-23T01:27:54.3571691Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3572108Z test_udf_remote_message_delay_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27481 2022-11-23T01:27:54.3572521Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27482 2022-11-23T01:27:54.3572838Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27483 2022-11-23T01:27:54.3573163Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27484 2022-11-23T01:27:54.3573614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3573943Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3574360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3574708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3575054Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbdh8rjfh 2022-11-23T01:27:54.3575448Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbdh8rjfh/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3575935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3576262Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3576689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3577017Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3577360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp58qi1opj 2022-11-23T01:27:54.3577766Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp58qi1opj/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3578241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3578602Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3579032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3579378Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3579705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp80feb_ja 2022-11-23T01:27:54.3580103Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp80feb_ja/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3580595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3580955Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3581370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3581716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3582066Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsgmghx7u 2022-11-23T01:27:54.3582457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsgmghx7u/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3582834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:27:54.3583177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:27:54.3583526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:27:54.3583859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:27:54.3584111Z ok (8.292s) 2022-11-23T01:27:54.3584214Z 2022-11-23T01:27:54.3584414Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3584641Z Ran 1 test in 8.293s 2022-11-23T01:27:54.3584753Z 2022-11-23T01:27:54.3584814Z OK 2022-11-23T01:27:54.3584907Z 2022-11-23T01:27:54.3584992Z Generating XML reports... 2022-11-23T01:27:54.3585465Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012613.xml 2022-11-23T01:27:54.3585995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3586330Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3586759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3587092Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3587440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxnr63g2q 2022-11-23T01:27:54.3587841Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxnr63g2q/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3588301Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-11-23T01:27:54.3588503Z 2022-11-23T01:27:54.3588562Z Running tests... 2022-11-23T01:27:54.3588867Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3589282Z test_udf_remote_message_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27656 2022-11-23T01:27:54.3589700Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27657 2022-11-23T01:27:54.3590017Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27658 2022-11-23T01:27:54.3590342Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27659 2022-11-23T01:27:54.3590790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3591110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3591575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3591924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3592362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbi603j5k 2022-11-23T01:27:54.3592751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbi603j5k/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3593242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3593572Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3594023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3594367Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3594714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkzn83ali 2022-11-23T01:27:54.3595111Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkzn83ali/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3595578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3595902Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3596326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3596668Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3596999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc4bl4a5i 2022-11-23T01:27:54.3597398Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc4bl4a5i/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3597879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3598188Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3598617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3598962Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3599306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq1u4cseq 2022-11-23T01:27:54.3599691Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq1u4cseq/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3600152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:27:54.3600838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:27:54.3601368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:27:54.3601796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:27:54.3602368Z [E request_callback_no_python.cpp:559] Received error while processing request type 261: false INTERNAL ASSERT FAILED at "/var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp":395, please report a bug to PyTorch. Expected OwnerRRef with id GloballyUniqueId(created_on=0, local_id=0) to be created. 2022-11-23T01:27:54.3602981Z Exception raised from getOwnerRRef at /var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp:395 (most recent call first): 2022-11-23T01:27:54.3603664Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x6b (0x7f9d28dde87b in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-11-23T01:27:54.3604491Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string, std::allocator > const&) + 0xce (0x7f9d28dda0de in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-11-23T01:27:54.3605310Z frame #2: c10::detail::torchInternalAssertFail(char const*, char const*, unsigned int, char const*, std::__cxx11::basic_string, std::allocator > const&) + 0x4e (0x7f9d28ddbfee in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-11-23T01:27:54.3606064Z frame #3: torch::distributed::rpc::RRefContext::getOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, bool) + 0x440 (0x7f9d2d830790 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-11-23T01:27:54.3607094Z frame #4: torch::distributed::rpc::RequestCallbackNoPython::assignOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, torch::distributed::rpc::GloballyUniqueId const&, c10::intrusive_ptr >) const + 0x5bc (0x7f9d2d82081c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-11-23T01:27:54.3608075Z frame #5: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector >) const + 0xcc (0x7f9d35bd052c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-11-23T01:27:54.3609286Z frame #6: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x194 (0x7f9d2d8250a4 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-11-23T01:27:54.3610405Z frame #7: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x65 (0x7f9d35bcfc35 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-11-23T01:27:54.3611066Z frame #8: + 0x49f84a3 (0x7f9d2d81e4a3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-11-23T01:27:54.3611775Z frame #9: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector >) const + 0x538 (0x7f9d2d81f038 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-11-23T01:27:54.3612584Z frame #10: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector >) const + 0x57 (0x7f9d2d819347 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-11-23T01:27:54.3613180Z frame #11: + 0x4a28012 (0x7f9d2d84e012 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-11-23T01:27:54.3613693Z frame #12: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f9d28dccbeb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-11-23T01:27:54.3614053Z frame #13: + 0xdbbf4 (0x7f9d28c4ebf4 in /opt/conda/lib/libstdc++.so.6) 2022-11-23T01:27:54.3614459Z frame #14: + 0x8609 (0x7f9d3da84609 in /lib/x86_64-linux-gnu/libpthread.so.0) 2022-11-23T01:27:54.3614827Z frame #15: clone + 0x43 (0x7f9d3d9a9133 in /lib/x86_64-linux-gnu/libc.so.6) 2022-11-23T01:27:54.3614988Z 2022-11-23T01:27:54.3615273Z [W tensorpipe_agent.cpp:624] RPC agent for worker0 won't send response to request #0 to worker0, as the agent is shutting down 2022-11-23T01:27:54.3615536Z ok (5.385s) 2022-11-23T01:27:54.3615636Z 2022-11-23T01:27:54.3615840Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3616080Z Ran 1 test in 5.386s 2022-11-23T01:27:54.3616193Z 2022-11-23T01:27:54.3616241Z OK 2022-11-23T01:27:54.3616330Z 2022-11-23T01:27:54.3616413Z Generating XML reports... 2022-11-23T01:27:54.3616930Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012623.xml 2022-11-23T01:27:54.3617476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3617794Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3618224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3618577Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3618912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf6rwl69g 2022-11-23T01:27:54.3619351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf6rwl69g/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3619813Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-11-23T01:27:54.3620013Z 2022-11-23T01:27:54.3620089Z Running tests... 2022-11-23T01:27:54.3620377Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3620792Z test_udf_remote_message_dropped_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27831 2022-11-23T01:27:54.3621209Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27832 2022-11-23T01:27:54.3621540Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27833 2022-11-23T01:27:54.3621855Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27834 2022-11-23T01:27:54.3622306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3622642Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3623057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3623404Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3623748Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbqqx0id_ 2022-11-23T01:27:54.3624151Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbqqx0id_/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3624622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3624949Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3625379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3625709Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3626051Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3r5dn6pd 2022-11-23T01:27:54.3626459Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3r5dn6pd/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3626945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3627263Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3627694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3628039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3628382Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9kp209k0 2022-11-23T01:27:54.3628767Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9kp209k0/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3629248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3629608Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3630026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3630160Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3630345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmgbsfeuv 2022-11-23T01:27:54.3630542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmgbsfeuv/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3630707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:27:54.3630916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:27:54.3631075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:27:54.3631237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:27:54.3631294Z ok (8.388s) 2022-11-23T01:27:54.3631314Z 2022-11-23T01:27:54.3631503Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3631578Z Ran 1 test in 8.388s 2022-11-23T01:27:54.3631584Z 2022-11-23T01:27:54.3631645Z OK 2022-11-23T01:27:54.3631650Z 2022-11-23T01:27:54.3631734Z Generating XML reports... 2022-11-23T01:27:54.3632157Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012630.xml 2022-11-23T01:27:54.3632443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3632573Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3632862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3632983Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3633172Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxlnmk0mi 2022-11-23T01:27:54.3633369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxlnmk0mi/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3633615Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-11-23T01:27:54.3633619Z 2022-11-23T01:27:54.3633695Z Running tests... 2022-11-23T01:27:54.3633893Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3634154Z test_udf_remote_message_dropped_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28006 2022-11-23T01:27:54.3634317Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28007 2022-11-23T01:27:54.3634474Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28008 2022-11-23T01:27:54.3634616Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28009 2022-11-23T01:27:54.3634889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3635013Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3635295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3635430Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3635614Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4vxyrmm6 2022-11-23T01:27:54.3635810Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4vxyrmm6/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3636085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3636193Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3636502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3636639Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3636824Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9fsr3__k 2022-11-23T01:27:54.3637017Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9fsr3__k/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3637289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3637414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3637723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3637857Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3638031Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsljy1ntc 2022-11-23T01:27:54.3638225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsljy1ntc/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3638493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3638616Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3638894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3639027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3639217Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2t0pnfb0 2022-11-23T01:27:54.3639411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2t0pnfb0/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3639562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:27:54.3639726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:27:54.3639888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:27:54.3640048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:27:54.3640116Z ok (8.389s) 2022-11-23T01:27:54.3640121Z 2022-11-23T01:27:54.3640320Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3640394Z Ran 1 test in 8.389s 2022-11-23T01:27:54.3640399Z 2022-11-23T01:27:54.3640460Z OK 2022-11-23T01:27:54.3640466Z 2022-11-23T01:27:54.3640549Z Generating XML reports... 2022-11-23T01:27:54.3641049Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012640.xml 2022-11-23T01:27:54.3641330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3641455Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3641738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3641874Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3642060Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3o4hn1k0 2022-11-23T01:27:54.3642256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3o4hn1k0/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3642507Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-11-23T01:27:54.3642514Z 2022-11-23T01:27:54.3642588Z Running tests... 2022-11-23T01:27:54.3642771Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3643069Z test_verify_backend_options (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28181 2022-11-23T01:27:54.3643229Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28182 2022-11-23T01:27:54.3643386Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28183 2022-11-23T01:27:54.3643540Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28184 2022-11-23T01:27:54.3643816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3643940Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3644261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3644382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3644568Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpybgib70s 2022-11-23T01:27:54.3644769Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpybgib70s/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3645042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3645166Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3645448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3645584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3645769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk39fruut 2022-11-23T01:27:54.3645967Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk39fruut/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3646225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3646349Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3646628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3646763Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3646947Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4y90o0_s 2022-11-23T01:27:54.3647141Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4y90o0_s/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3647411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3647535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3647799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3647935Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3648118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbtvz5efd 2022-11-23T01:27:54.3648314Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbtvz5efd/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3648478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:27:54.3648639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:27:54.3648801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:27:54.3648967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:27:54.3649021Z ok (8.288s) 2022-11-23T01:27:54.3649041Z 2022-11-23T01:27:54.3649227Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3649302Z Ran 1 test in 8.288s 2022-11-23T01:27:54.3649308Z 2022-11-23T01:27:54.3649402Z OK 2022-11-23T01:27:54.3649408Z 2022-11-23T01:27:54.3649494Z Generating XML reports... 2022-11-23T01:27:54.3649848Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012650.xml 2022-11-23T01:27:54.3650120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3650244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3650523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3650673Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3650859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq0fbif11 2022-11-23T01:27:54.3651054Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq0fbif11/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3651303Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-11-23T01:27:54.3651308Z 2022-11-23T01:27:54.3651379Z Running tests... 2022-11-23T01:27:54.3651573Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3651821Z test_remote_timeout_to_here_in_jit (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28356 2022-11-23T01:27:54.3651980Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28357 2022-11-23T01:27:54.3652133Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28358 2022-11-23T01:27:54.3652276Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28359 2022-11-23T01:27:54.3652546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3652669Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3652947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3653080Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3653263Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp01efc7mn 2022-11-23T01:27:54.3653459Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp01efc7mn/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3653730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3653842Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3654119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3654253Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3654440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpse4zdwwi 2022-11-23T01:27:54.3654635Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpse4zdwwi/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3654903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3655025Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3655302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3655436Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3655609Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpygyz13em 2022-11-23T01:27:54.3655803Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpygyz13em/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3656097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3656223Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3656499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3656633Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3656814Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnn_sw_wn 2022-11-23T01:27:54.3657004Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnn_sw_wn/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3657183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:27:54.3657347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:27:54.3657511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:27:54.3657671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:27:54.3657737Z ok (8.390s) 2022-11-23T01:27:54.3657743Z 2022-11-23T01:27:54.3657939Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3658013Z Ran 1 test in 8.391s 2022-11-23T01:27:54.3658018Z 2022-11-23T01:27:54.3658078Z OK 2022-11-23T01:27:54.3658083Z 2022-11-23T01:27:54.3658164Z Generating XML reports... 2022-11-23T01:27:54.3658513Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20221123012700.xml 2022-11-23T01:27:54.3658788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3658911Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3659193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3659329Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3659513Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqo8lhc9_ 2022-11-23T01:27:54.3659708Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqo8lhc9_/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3659952Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-11-23T01:27:54.3659957Z 2022-11-23T01:27:54.3660031Z Running tests... 2022-11-23T01:27:54.3660212Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3660458Z test_rref_timeout_pickle_in_jit (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28531 2022-11-23T01:27:54.3660612Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28532 2022-11-23T01:27:54.3660765Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28533 2022-11-23T01:27:54.3660917Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28534 2022-11-23T01:27:54.3661187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3661310Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3661589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3661711Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3661899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprs55zaam 2022-11-23T01:27:54.3662094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprs55zaam/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3662388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3662512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3662792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3662925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3663108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphosm_yyf 2022-11-23T01:27:54.3663301Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphosm_yyf/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3663597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3663720Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3664000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3664134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3664317Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8dmr9abv 2022-11-23T01:27:54.3664511Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8dmr9abv/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3664783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3664903Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3665169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3665308Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3665489Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6_h_r1c3 2022-11-23T01:27:54.3665684Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6_h_r1c3/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3665846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:27:54.3666004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:27:54.3666165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:27:54.3666328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:27:54.3666396Z ok (8.289s) 2022-11-23T01:27:54.3666401Z 2022-11-23T01:27:54.3666584Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3666662Z Ran 1 test in 8.290s 2022-11-23T01:27:54.3666667Z 2022-11-23T01:27:54.3666729Z OK 2022-11-23T01:27:54.3666734Z 2022-11-23T01:27:54.3666822Z Generating XML reports... 2022-11-23T01:27:54.3667191Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20221123012711.xml 2022-11-23T01:27:54.3667462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3667585Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3667863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3667983Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3668167Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6hgd9uxl 2022-11-23T01:27:54.3668364Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6hgd9uxl/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3668605Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-11-23T01:27:54.3668609Z 2022-11-23T01:27:54.3668683Z Running tests... 2022-11-23T01:27:54.3668907Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3669162Z test_rref_timeout_pickle_script_func (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28706 2022-11-23T01:27:54.3669320Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28707 2022-11-23T01:27:54.3669476Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28708 2022-11-23T01:27:54.3669616Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28709 2022-11-23T01:27:54.3669889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3670046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3670329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3670464Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3670646Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy_bspjnk 2022-11-23T01:27:54.3670841Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy_bspjnk/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3671111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3671219Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3671499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3671632Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3671814Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptbmidzt2 2022-11-23T01:27:54.3672008Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptbmidzt2/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3672362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3672486Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3672768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3672905Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3673076Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprpch4v3h 2022-11-23T01:27:54.3673274Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprpch4v3h/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3673541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3673665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3673951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3674087Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3674274Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgzevm5g8 2022-11-23T01:27:54.3674469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgzevm5g8/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3674634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:27:54.3674785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:27:54.3674950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:27:54.3675112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:27:54.3675180Z ok (8.394s) 2022-11-23T01:27:54.3675217Z 2022-11-23T01:27:54.3675414Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3675488Z Ran 1 test in 8.394s 2022-11-23T01:27:54.3675493Z 2022-11-23T01:27:54.3675556Z OK 2022-11-23T01:27:54.3675560Z 2022-11-23T01:27:54.3675644Z Generating XML reports... 2022-11-23T01:27:54.3675993Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20221123012721.xml 2022-11-23T01:27:54.3676264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3676388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3676707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3676840Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3677025Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5pms6c2e 2022-11-23T01:27:54.3677219Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5pms6c2e/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3677460Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-11-23T01:27:54.3677466Z 2022-11-23T01:27:54.3677537Z Running tests... 2022-11-23T01:27:54.3677721Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3677966Z test_rref_to_here_timeout_in_jit (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28881 2022-11-23T01:27:54.3678124Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28882 2022-11-23T01:27:54.3678277Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28883 2022-11-23T01:27:54.3678428Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28884 2022-11-23T01:27:54.3678701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3678823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3679100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3679220Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3679403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr7p0wil9 2022-11-23T01:27:54.3679598Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr7p0wil9/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3679874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3679998Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3680277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3680410Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3680734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpctmikko1 2022-11-23T01:27:54.3680936Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpctmikko1/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3681196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3681321Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3681604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3681741Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3681981Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxd93rglf 2022-11-23T01:27:54.3682177Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxd93rglf/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3682450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3682575Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3682839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3682975Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3683160Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc59hnj1i 2022-11-23T01:27:54.3683389Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc59hnj1i/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3683556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:27:54.3683720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:27:54.3683882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:27:54.3684045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:27:54.3684115Z ok (4.490s) 2022-11-23T01:27:54.3684120Z 2022-11-23T01:27:54.3684303Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3684378Z Ran 1 test in 4.490s 2022-11-23T01:27:54.3684384Z 2022-11-23T01:27:54.3684444Z OK 2022-11-23T01:27:54.3684449Z 2022-11-23T01:27:54.3684535Z Generating XML reports... 2022-11-23T01:27:54.3684899Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20221123012731.xml 2022-11-23T01:27:54.3685173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3685300Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3685582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3685704Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3685889Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp27i52nte 2022-11-23T01:27:54.3686084Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp27i52nte/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3686329Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-11-23T01:27:54.3686337Z 2022-11-23T01:27:54.3686409Z Running tests... 2022-11-23T01:27:54.3686604Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3686846Z test_timeout_in_python (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29056 2022-11-23T01:27:54.3687003Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29057 2022-11-23T01:27:54.3687157Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29058 2022-11-23T01:27:54.3687296Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29059 2022-11-23T01:27:54.3687569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3687693Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3687973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3688109Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3688294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu4jruzbm 2022-11-23T01:27:54.3688522Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu4jruzbm/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3688798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3688919Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3689182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3689317Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3689501Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkzd5z_tl 2022-11-23T01:27:54.3689723Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkzd5z_tl/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3689996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3690120Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3690397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3690531Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3690703Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8vl05mgh 2022-11-23T01:27:54.3690898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8vl05mgh/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3691171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3691296Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3691577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3691711Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3691900Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppat219_u 2022-11-23T01:27:54.3692094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppat219_u/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3692258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:27:54.3692409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:27:54.3692571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:27:54.3692734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:27:54.3692803Z ok (6.990s) 2022-11-23T01:27:54.3692808Z 2022-11-23T01:27:54.3693004Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3693080Z Ran 1 test in 6.990s 2022-11-23T01:27:54.3693084Z 2022-11-23T01:27:54.3693147Z OK 2022-11-23T01:27:54.3693154Z 2022-11-23T01:27:54.3693237Z Generating XML reports... 2022-11-23T01:27:54.3693592Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20221123012737.xml 2022-11-23T01:27:54.3693865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3693989Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3694268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3694404Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3694589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpee_8uk0e 2022-11-23T01:27:54.3694781Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpee_8uk0e/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3695054Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent 2022-11-23T01:27:54.3695060Z 2022-11-23T01:27:54.3695133Z Running tests... 2022-11-23T01:27:54.3695317Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3695567Z test_timeout_in_torchscript_function (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29231 2022-11-23T01:27:54.3695725Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29232 2022-11-23T01:27:54.3695880Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29233 2022-11-23T01:27:54.3696063Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29234 2022-11-23T01:27:54.3696335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3696458Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3696736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3696858Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3697043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn0dedfbq 2022-11-23T01:27:54.3697240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn0dedfbq/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3697514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3697638Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3697919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3698053Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3698238Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprvvg26mx 2022-11-23T01:27:54.3698507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3698615Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3698894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3699027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3699213Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppvrbaymo 2022-11-23T01:27:54.3699411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprvvg26mx/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3699605Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppvrbaymo/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3699882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:27:54.3700003Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:27:54.3700268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:27:54.3700401Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:27:54.3700585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4youtsrt 2022-11-23T01:27:54.3700779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4youtsrt/_remote_module_non_scriptable.py 2022-11-23T01:27:54.3700943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:27:54.3701101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:27:54.3701262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:27:54.3701456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:27:54.3701524Z ok (6.889s) 2022-11-23T01:27:54.3701530Z 2022-11-23T01:27:54.3701712Z ---------------------------------------------------------------------- 2022-11-23T01:27:54.3701788Z Ran 1 test in 6.890s 2022-11-23T01:27:54.3701793Z 2022-11-23T01:27:54.3701853Z OK 2022-11-23T01:27:54.3701857Z 2022-11-23T01:27:54.3701940Z Generating XML reports... 2022-11-23T01:27:54.3702304Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20221123012746.xml 2022-11-23T01:27:54.3702336Z 2022-11-23T01:27:54.3702727Z ##[endgroup] 2022-11-23T01:27:54.3703084Z FINISHED PRINTING LOG FILE of distributed/rpc/test_faulty_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_faulty_agent_0h0w8bg_) 2022-11-23T01:27:54.3703090Z 2022-11-23T01:27:54.3703305Z Running distributed/rpc/cuda/test_tensorpipe_agent ... [2022-11-23 01:27:54.313251] 2022-11-23T01:27:54.3703694Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/cuda/test_tensorpipe_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:27:54.313769] 2022-11-23T01:34:53.7336342Z 2022-11-23T01:34:53.7336763Z Expand the folded group to see the log file of distributed/rpc/cuda/test_tensorpipe_agent 2022-11-23T01:34:53.7338289Z ##[group]PRINTING LOG FILE of distributed/rpc/cuda/test_tensorpipe_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-cuda-test_tensorpipe_agent_myzf8yej) 2022-11-23T01:34:53.7341950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb1gmx0lj 2022-11-23T01:34:53.7342580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb1gmx0lj/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7343298Z ]> 2022-11-23T01:34:53.7343955Z test_ddp_dist_autograd_local_vs_remote_gpu (__main__.TensorPipeCudaDdpComparisonTest) 2022-11-23T01:34:53.7370728Z , <__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_to_cpu_continuation>, <__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_to_cpu_continuation_gpu_root>]> 2022-11-23T01:34:53.7371832Z test_gpu_simple (__main__.TensorPipeCudaDistAutogradTest) 2022-11-23T01:34:53.7372384Z test_gpu_to_cpu_continuation (__main__.TensorPipeCudaDistAutogradTest) 2022-11-23T01:34:53.7372926Z test_gpu_to_cpu_continuation_gpu_root (__main__.TensorPipeCudaDistAutogradTest) 2022-11-23T01:34:53.7373885Z , <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_input_moved_to_cuda_device_script>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_invalid_devices>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_valid_device>]> 2022-11-23T01:34:53.7374785Z test_input_moved_to_cuda_device (__main__.TensorPipeCudaRemoteModuleTest) 2022-11-23T01:34:53.7375249Z test_input_moved_to_cuda_device_script (__main__.TensorPipeCudaRemoteModuleTest) 2022-11-23T01:34:53.7375715Z test_invalid_devices (__main__.TensorPipeCudaRemoteModuleTest) 2022-11-23T01:34:53.7376132Z test_valid_device (__main__.TensorPipeCudaRemoteModuleTest) 2022-11-23T01:34:53.7376638Z ]> 2022-11-23T01:34:53.7377100Z test_profiler_remote_cuda (__main__.TensorPipeCudaRpcTest) 2022-11-23T01:34:53.7378791Z , <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_except_last>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_never>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_never_find_unused>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_always>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_except_last>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_never>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_never_find_unused>]> 2022-11-23T01:34:53.7380178Z test_basic_gloo_ckpt_always (__main__.TensorPipePipeWithDDPTest) 2022-11-23T01:34:53.7380685Z test_basic_gloo_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) 2022-11-23T01:34:53.7381265Z test_basic_gloo_ckpt_never (__main__.TensorPipePipeWithDDPTest) 2022-11-23T01:34:53.7381922Z test_basic_gloo_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) 2022-11-23T01:34:53.7382446Z test_basic_nccl_ckpt_always (__main__.TensorPipePipeWithDDPTest) 2022-11-23T01:34:53.7382928Z test_basic_nccl_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) 2022-11-23T01:34:53.7383421Z test_basic_nccl_ckpt_never (__main__.TensorPipePipeWithDDPTest) 2022-11-23T01:34:53.7383898Z test_basic_nccl_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) 2022-11-23T01:34:53.7399502Z , <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_async_execution_with_cuda_future>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_callback_changes_devices>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_custom_class_with_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_list_with_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_list_with_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_int>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_str>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_not_cuda>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_modify_tensor_inplace>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_replace_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_value_on_bad_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_multi>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_nested>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_nested_multi>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu_to_gpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu_to_gpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_default_to_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_6>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_7>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_8>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_6>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_7>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_8>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_non_default_to_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_to_cpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_to_cpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_in_options>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_max_local_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_max_remote_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_min_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_many_to_one>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_loop>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_not_timeout>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_remote>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_remote_response>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_response>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_response_loop>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_multi_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_multi_gpu_self>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_one_to_many>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_remote>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_return_to_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_return_to_gpu_self>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_wrong_worker_name>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_mismatch>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_devices_option_mismatch>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_devices_option_mismatch_reverse>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_with_unpickleable_attributes>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_tensor_view_as_return_value>]> 2022-11-23T01:34:53.7414558Z test_async_execution_nested_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7415082Z test_async_execution_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7415610Z test_cuda_future_callback_changes_devices (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7416133Z test_cuda_future_can_extract_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7416671Z test_cuda_future_can_extract_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7417205Z test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7417754Z test_cuda_future_can_extract_custom_class_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7418394Z test_cuda_future_can_extract_list_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7418899Z test_cuda_future_can_extract_list_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7419382Z test_cuda_future_device_as_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7419871Z test_cuda_future_device_as_int (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7420384Z test_cuda_future_device_as_str (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7420948Z test_cuda_future_device_not_cuda (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7421460Z test_cuda_future_modify_tensor_inplace (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7422015Z test_cuda_future_replace_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7422509Z test_cuda_future_value_on_bad_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7423037Z test_custom_stream (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7423527Z test_custom_stream_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7424071Z test_custom_stream_nested (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7424643Z test_custom_stream_nested_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7425209Z test_device_map_cpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7425762Z test_device_map_cpu_to_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7426257Z test_device_map_cpu_to_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7426736Z test_device_map_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7427368Z test_device_map_gpu_default_to_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7427856Z test_device_map_gpu_mixed_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7428333Z test_device_map_gpu_mixed_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7428833Z test_device_map_gpu_mixed_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7429314Z test_device_map_gpu_mixed_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7429731Z test_device_map_gpu_mixed_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7430186Z test_device_map_gpu_mixed_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7430761Z test_device_map_gpu_mixed_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7431234Z test_device_map_gpu_mixed_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7431804Z test_device_map_gpu_mixed_self_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7432416Z test_device_map_gpu_mixed_self_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7432957Z test_device_map_gpu_mixed_self_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7433403Z test_device_map_gpu_mixed_self_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7433855Z test_device_map_gpu_mixed_self_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7434327Z test_device_map_gpu_mixed_self_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7434810Z test_device_map_gpu_mixed_self_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7435303Z test_device_map_gpu_mixed_self_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7435806Z test_device_map_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7436330Z test_device_map_gpu_non_default_to_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7436833Z test_device_map_gpu_to_cpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7437323Z test_device_map_gpu_to_cpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7437754Z test_device_maps_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7438205Z test_device_maps_in_options (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7438689Z test_device_maps_invalid_max_local_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7439270Z test_device_maps_invalid_max_remote_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7439713Z test_device_maps_invalid_min_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7440239Z test_device_maps_many_to_one (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7441024Z test_device_maps_missing_config (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7441524Z test_device_maps_missing_config_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7442042Z test_device_maps_missing_config_not_timeout (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7442495Z test_device_maps_missing_config_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7443061Z test_device_maps_missing_config_remote_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7443577Z test_device_maps_missing_config_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7444171Z test_device_maps_missing_config_response_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7444717Z test_device_maps_multi_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7445213Z test_device_maps_multi_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7445725Z test_device_maps_one_to_many (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7446295Z test_device_maps_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7446802Z test_device_maps_return_to_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7447352Z test_device_maps_return_to_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7447847Z test_device_maps_wrong_worker_name (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7448349Z test_device_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7448832Z test_devices_option_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7449268Z test_devices_option_mismatch_reverse (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7449832Z test_owner_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7450269Z test_owner_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7450657Z test_owner_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7451029Z test_owner_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7451405Z test_rref_as_arg_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7451779Z test_rref_as_arg_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7452146Z test_rref_as_arg_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7452498Z test_rref_as_arg_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7452859Z test_rref_as_arg_synchronization5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7453230Z test_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7453594Z test_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7453968Z test_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7454344Z test_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7454713Z test_rref_to_here_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7455069Z test_rref_to_here_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7455438Z test_rref_to_here_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7455805Z test_rref_to_here_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7456165Z test_rref_with_unpickleable_attributes (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7456541Z test_tensor_view_as_return_value (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-11-23T01:34:53.7457236Z , <__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_dist_autograd_sync_streams>, <__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_gradients_synchronizations>]> 2022-11-23T01:34:53.7457929Z test_device_maps_backward_pass (__main__.TensorPipeTensorPipeCudaDistAutogradTest) 2022-11-23T01:34:53.7458302Z test_dist_autograd_sync_streams (__main__.TensorPipeTensorPipeCudaDistAutogradTest) 2022-11-23T01:34:53.7474070Z test_gradients_synchronizations (__main__.TensorPipeTensorPipeCudaDistAutogradTest) 2022-11-23T01:34:53.7475092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7475680Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7476512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7477163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7477866Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmu_ej8e6 2022-11-23T01:34:53.7478464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmu_ej8e6/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7478963Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.7479183Z 2022-11-23T01:34:53.7479255Z Running tests... 2022-11-23T01:34:53.7479546Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7479952Z test_ddp_dist_autograd_local_vs_remote_gpu (__main__.TensorPipeCudaDdpComparisonTest) ... skip: c10d was not compiled with the NCCL backend (0.002s) 2022-11-23T01:34:53.7480263Z 2022-11-23T01:34:53.7480458Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7480922Z Ran 1 test in 0.002s 2022-11-23T01:34:53.7481082Z 2022-11-23T01:34:53.7481155Z OK (skipped=1) 2022-11-23T01:34:53.7481309Z 2022-11-23T01:34:53.7481447Z Generating XML reports... 2022-11-23T01:34:53.7482337Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDdpComparisonTest-20221123012757.xml 2022-11-23T01:34:53.7483199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7483642Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7484200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7484703Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7485250Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4gspy4_m 2022-11-23T01:34:53.7485850Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4gspy4_m/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7486348Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.7486566Z 2022-11-23T01:34:53.7486626Z Running tests... 2022-11-23T01:34:53.7486928Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7487333Z test_gpu_simple (__main__.TensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29430 2022-11-23T01:34:53.7487743Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29431 2022-11-23T01:34:53.7488059Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29432 2022-11-23T01:34:53.7488380Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29433 2022-11-23T01:34:53.7488838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7489222Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7489640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7489989Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7490331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp_ac3v46 2022-11-23T01:34:53.7490811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7491123Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7491549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7491896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7492230Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7x19ywh1 2022-11-23T01:34:53.7492731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp_ac3v46/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7493141Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7x19ywh1/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7493632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7493947Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7494370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7494718Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7495061Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3bfyss53 2022-11-23T01:34:53.7495495Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3bfyss53/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7495984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7496309Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7496721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7497064Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7512366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpryckv330 2022-11-23T01:34:53.7513272Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpryckv330/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7514097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.7514858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.7515608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.7516372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.7516986Z skip: Need at least 1 CUDA device (2.181s) 2022-11-23T01:34:53.7517248Z 2022-11-23T01:34:53.7517752Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7518263Z Ran 1 test in 2.182s 2022-11-23T01:34:53.7518497Z 2022-11-23T01:34:53.7518649Z OK (skipped=1) 2022-11-23T01:34:53.7518869Z 2022-11-23T01:34:53.7519023Z Generating XML reports... 2022-11-23T01:34:53.7520165Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221123012758.xml 2022-11-23T01:34:53.7521463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7522076Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7522621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7523243Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7523847Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpugs5lyd7 2022-11-23T01:34:53.7524472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpugs5lyd7/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7525390Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.7525793Z 2022-11-23T01:34:53.7525927Z Running tests... 2022-11-23T01:34:53.7526480Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7527232Z test_gpu_to_cpu_continuation (__main__.TensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29485 2022-11-23T01:34:53.7528009Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29486 2022-11-23T01:34:53.7528789Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29487 2022-11-23T01:34:53.7529411Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29488 2022-11-23T01:34:53.7530248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7530868Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7531665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7532292Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7533000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3n4lrk6d 2022-11-23T01:34:53.7533910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7534517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7535302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7535940Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7536572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7h7k6_cu 2022-11-23T01:34:53.7537303Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3n4lrk6d/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7538047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7h7k6_cu/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7538958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7539568Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7540362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7541000Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7541632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1d6tijko 2022-11-23T01:34:53.7542375Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1d6tijko/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7543271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7543874Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7544673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7545293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7545926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps5gbx5ll 2022-11-23T01:34:53.7546679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps5gbx5ll/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7547378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.7548013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.7548648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.7549289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.7549799Z skip: Need at least 1 CUDA device (2.184s) 2022-11-23T01:34:53.7550044Z 2022-11-23T01:34:53.7550406Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7550844Z Ran 1 test in 2.185s 2022-11-23T01:34:53.7551047Z 2022-11-23T01:34:53.7551178Z OK (skipped=1) 2022-11-23T01:34:53.7551351Z 2022-11-23T01:34:53.7551501Z Generating XML reports... 2022-11-23T01:34:53.7552607Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221123012803.xml 2022-11-23T01:34:53.7553716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7554314Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7555121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7555764Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7556405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ne4izaq 2022-11-23T01:34:53.7557182Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ne4izaq/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7558077Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.7558480Z 2022-11-23T01:34:53.7558614Z Running tests... 2022-11-23T01:34:53.7559179Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7559942Z test_gpu_to_cpu_continuation_gpu_root (__main__.TensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29540 2022-11-23T01:34:53.7560822Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29541 2022-11-23T01:34:53.7561449Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29542 2022-11-23T01:34:53.7562049Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29543 2022-11-23T01:34:53.7562905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7563512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7564315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7564936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7565572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmped3zyles 2022-11-23T01:34:53.7566323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmped3zyles/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7567239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7567823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7568676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7569321Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7569957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpix73cqk3 2022-11-23T01:34:53.7570644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpix73cqk3/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7571519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7572069Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7572810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7573355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7573940Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu3c4quov 2022-11-23T01:34:53.7574622Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu3c4quov/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7575470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7576142Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7576889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7577465Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7578019Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbg33vh5i 2022-11-23T01:34:53.7578662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbg33vh5i/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7579276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.7579852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.7580481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.7581045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.7581493Z skip: Need at least 1 CUDA device (2.289s) 2022-11-23T01:34:53.7581704Z 2022-11-23T01:34:53.7582095Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7582489Z Ran 1 test in 2.289s 2022-11-23T01:34:53.7582611Z 2022-11-23T01:34:53.7582721Z OK (skipped=1) 2022-11-23T01:34:53.7582864Z 2022-11-23T01:34:53.7582936Z Generating XML reports... 2022-11-23T01:34:53.7583450Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221123012807.xml 2022-11-23T01:34:53.7584028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7584361Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7584775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7585122Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7585465Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpur076dl4 2022-11-23T01:34:53.7585865Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpur076dl4/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7586331Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.7586548Z 2022-11-23T01:34:53.7586619Z Running tests... 2022-11-23T01:34:53.7586921Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7587457Z test_input_moved_to_cuda_device (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29595 2022-11-23T01:34:53.7587879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29596 2022-11-23T01:34:53.7588326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7588658Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7589070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7589411Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7589753Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa1oufyfm 2022-11-23T01:34:53.7590155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa1oufyfm/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7590624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7590951Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7591377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7591769Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7592117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq9je7oah 2022-11-23T01:34:53.7592599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq9je7oah/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7592973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.7593305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.7593585Z skip: Need at least 1 CUDA device (2.076s) 2022-11-23T01:34:53.7593719Z 2022-11-23T01:34:53.7593954Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7594180Z Ran 1 test in 2.076s 2022-11-23T01:34:53.7594291Z 2022-11-23T01:34:53.7594362Z OK (skipped=1) 2022-11-23T01:34:53.7594467Z 2022-11-23T01:34:53.7594553Z Generating XML reports... 2022-11-23T01:34:53.7595064Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221123012811.xml 2022-11-23T01:34:53.7595627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7595959Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7596384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7596727Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7597055Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy5ro83vi 2022-11-23T01:34:53.7597458Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy5ro83vi/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7597936Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.7598150Z 2022-11-23T01:34:53.7598212Z Running tests... 2022-11-23T01:34:53.7598509Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7598934Z test_input_moved_to_cuda_device_script (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29628 2022-11-23T01:34:53.7599359Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29629 2022-11-23T01:34:53.7599799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7600125Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7600552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7601067Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7601403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5f67nfew 2022-11-23T01:34:53.7601889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7602222Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7602878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7603408Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7603942Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdmblec2f 2022-11-23T01:34:53.7604675Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5f67nfew/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7605136Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdmblec2f/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7605511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.7605940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.7606216Z skip: Need at least 1 CUDA device (2.078s) 2022-11-23T01:34:53.7606351Z 2022-11-23T01:34:53.7606557Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7606796Z Ran 1 test in 2.079s 2022-11-23T01:34:53.7606908Z 2022-11-23T01:34:53.7606978Z OK (skipped=1) 2022-11-23T01:34:53.7607071Z 2022-11-23T01:34:53.7607153Z Generating XML reports... 2022-11-23T01:34:53.7607665Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221123012815.xml 2022-11-23T01:34:53.7608287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7608628Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7609267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7609717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7610066Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph8mlvx2m 2022-11-23T01:34:53.7610594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph8mlvx2m/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7611363Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.7611716Z 2022-11-23T01:34:53.7611829Z Running tests... 2022-11-23T01:34:53.7612366Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7613048Z test_invalid_devices (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29661 2022-11-23T01:34:53.7613806Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29662 2022-11-23T01:34:53.7614436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7614759Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7615240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7615588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7615979Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8m2u4sgq 2022-11-23T01:34:53.7616575Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8m2u4sgq/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7617393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7617944Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7618379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7618710Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7619050Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5atdfglq 2022-11-23T01:34:53.7619726Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5atdfglq/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7620358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.7620706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.7620992Z skip: Need at least 1 CUDA device (2.074s) 2022-11-23T01:34:53.7621136Z 2022-11-23T01:34:53.7621460Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7621826Z Ran 1 test in 2.074s 2022-11-23T01:34:53.7621996Z 2022-11-23T01:34:53.7622107Z OK (skipped=1) 2022-11-23T01:34:53.7622363Z 2022-11-23T01:34:53.7622505Z Generating XML reports... 2022-11-23T01:34:53.7623403Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221123012819.xml 2022-11-23T01:34:53.7624481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7624981Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7625409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7625741Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7626138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpebke3asm 2022-11-23T01:34:53.7626542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpebke3asm/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7627025Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.7627230Z 2022-11-23T01:34:53.7627301Z Running tests... 2022-11-23T01:34:53.7627603Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7628155Z test_valid_device (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29694 2022-11-23T01:34:53.7628680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29695 2022-11-23T01:34:53.7629132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7629463Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7629887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7630223Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7630571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfzwlgnh1 2022-11-23T01:34:53.7630973Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfzwlgnh1/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7631471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7631967Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7632780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7633347Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7633889Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkvefasfk 2022-11-23T01:34:53.7634530Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkvefasfk/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7635212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.7635844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.7636339Z skip: Need at least 1 CUDA device (2.073s) 2022-11-23T01:34:53.7636580Z 2022-11-23T01:34:53.7636910Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7637164Z Ran 1 test in 2.073s 2022-11-23T01:34:53.7637274Z 2022-11-23T01:34:53.7637332Z OK (skipped=1) 2022-11-23T01:34:53.7637437Z 2022-11-23T01:34:53.7637519Z Generating XML reports... 2022-11-23T01:34:53.7638031Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221123012822.xml 2022-11-23T01:34:53.7638612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7639004Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7639438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7639781Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7640112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpomfr83sy 2022-11-23T01:34:53.7640515Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpomfr83sy/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7641125Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.7641421Z 2022-11-23T01:34:53.7641494Z Running tests... 2022-11-23T01:34:53.7641783Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7642185Z test_profiler_remote_cuda (__main__.TensorPipeCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29727 2022-11-23T01:34:53.7642589Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29728 2022-11-23T01:34:53.7642919Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29729 2022-11-23T01:34:53.7643231Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29730 2022-11-23T01:34:53.7643683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7644016Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7644432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7644780Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7645124Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpznmi367y 2022-11-23T01:34:53.7645524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpznmi367y/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7646003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7646329Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7646752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7647087Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7647427Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_i5zshgp 2022-11-23T01:34:53.7647903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7648230Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7648641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7648983Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7649325Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmputkv6j68 2022-11-23T01:34:53.7649724Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_i5zshgp/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7650113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmputkv6j68/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7650598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7650920Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7651331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7651672Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7652055Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_yi_72to 2022-11-23T01:34:53.7652452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_yi_72to/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7652813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.7653158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.7653502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.7653832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.7654151Z skip: Need at least 2 CUDA devices (2.284s) 2022-11-23T01:34:53.7654287Z 2022-11-23T01:34:53.7654486Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7654722Z Ran 1 test in 2.284s 2022-11-23T01:34:53.7654821Z 2022-11-23T01:34:53.7654891Z OK (skipped=1) 2022-11-23T01:34:53.7654998Z 2022-11-23T01:34:53.7655082Z Generating XML reports... 2022-11-23T01:34:53.7655563Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRpcTest-20221123012826.xml 2022-11-23T01:34:53.7656104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7656432Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7656861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7657206Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7657540Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0vx56gev 2022-11-23T01:34:53.7657938Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0vx56gev/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7658422Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.7658636Z 2022-11-23T01:34:53.7658707Z Running tests... 2022-11-23T01:34:53.7658990Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7659398Z test_basic_gloo_ckpt_always (__main__.TensorPipePipeWithDDPTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29782 2022-11-23T01:34:53.7659806Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29783 2022-11-23T01:34:53.7660243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7660574Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7660998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7661347Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7661676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0fm_p6_3 2022-11-23T01:34:53.7662071Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0fm_p6_3/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7662551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7662877Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7663289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7663636Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7663978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfa38hygw 2022-11-23T01:34:53.7664364Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfa38hygw/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7664767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.7665116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.7665400Z skip: Need at least 4 CUDA devices (2.073s) 2022-11-23T01:34:53.7665522Z 2022-11-23T01:34:53.7665719Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7665958Z Ran 1 test in 2.073s 2022-11-23T01:34:53.7666071Z 2022-11-23T01:34:53.7666141Z OK (skipped=1) 2022-11-23T01:34:53.7666247Z 2022-11-23T01:34:53.7666318Z Generating XML reports... 2022-11-23T01:34:53.7666814Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123012830.xml 2022-11-23T01:34:53.7667418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7667749Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7668166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7668511Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7668852Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeyzxylcz 2022-11-23T01:34:53.7669240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeyzxylcz/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7669717Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.7669935Z 2022-11-23T01:34:53.7670007Z Running tests... 2022-11-23T01:34:53.7670307Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7670709Z test_basic_gloo_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29815 2022-11-23T01:34:53.7671123Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29816 2022-11-23T01:34:53.7671576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7671907Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7672434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7672783Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7673126Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7cej06g0 2022-11-23T01:34:53.7673515Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7cej06g0/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7674002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7674332Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7674757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7675088Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7675430Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp02edlnu4 2022-11-23T01:34:53.7675828Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp02edlnu4/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7676200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.7676539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.7676823Z skip: Need at least 4 CUDA devices (2.075s) 2022-11-23T01:34:53.7676956Z 2022-11-23T01:34:53.7677152Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7677420Z Ran 1 test in 2.076s 2022-11-23T01:34:53.7677535Z 2022-11-23T01:34:53.7677606Z OK (skipped=1) 2022-11-23T01:34:53.7677711Z 2022-11-23T01:34:53.7677795Z Generating XML reports... 2022-11-23T01:34:53.7678279Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123012834.xml 2022-11-23T01:34:53.7678844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7679172Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7679604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7679967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7680310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxt3_4ey5 2022-11-23T01:34:53.7680837Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxt3_4ey5/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7681327Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.7681532Z 2022-11-23T01:34:53.7681605Z Running tests... 2022-11-23T01:34:53.7681906Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7682316Z test_basic_gloo_ckpt_never (__main__.TensorPipePipeWithDDPTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29848 2022-11-23T01:34:53.7682712Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29849 2022-11-23T01:34:53.7683169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7683501Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7684096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7684612Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7685208Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw2akevzz 2022-11-23T01:34:53.7685928Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw2akevzz/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7686722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7687231Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7687925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7688497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7689055Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnbva8_s7 2022-11-23T01:34:53.7689701Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnbva8_s7/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7690323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.7690911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.7691355Z skip: Need at least 4 CUDA devices (2.081s) 2022-11-23T01:34:53.7691577Z 2022-11-23T01:34:53.7691907Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7692297Z Ran 1 test in 2.081s 2022-11-23T01:34:53.7692480Z 2022-11-23T01:34:53.7692604Z OK (skipped=1) 2022-11-23T01:34:53.7692761Z 2022-11-23T01:34:53.7692898Z Generating XML reports... 2022-11-23T01:34:53.7693723Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123012838.xml 2022-11-23T01:34:53.7694762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7695279Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7695983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7696551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7697109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu9plt9ab 2022-11-23T01:34:53.7697735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu9plt9ab/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7698513Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.7698925Z 2022-11-23T01:34:53.7699049Z Running tests... 2022-11-23T01:34:53.7699533Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7700188Z test_basic_gloo_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29881 2022-11-23T01:34:53.7700942Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29882 2022-11-23T01:34:53.7701771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7702350Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7702931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7703282Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7703782Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpokcxoxdo 2022-11-23T01:34:53.7704407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpokcxoxdo/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7705198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7705695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7706363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7706879Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7707434Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7054tngn 2022-11-23T01:34:53.7708025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7054tngn/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7708598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.7709141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.7709582Z skip: Need at least 4 CUDA devices (2.072s) 2022-11-23T01:34:53.7709789Z 2022-11-23T01:34:53.7710105Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7710443Z Ran 1 test in 2.073s 2022-11-23T01:34:53.7710614Z 2022-11-23T01:34:53.7710739Z OK (skipped=1) 2022-11-23T01:34:53.7710899Z 2022-11-23T01:34:53.7711029Z Generating XML reports... 2022-11-23T01:34:53.7711824Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123012842.xml 2022-11-23T01:34:53.7712806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7713326Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7714032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7714584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7715199Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpahc0u7pb 2022-11-23T01:34:53.7718380Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpahc0u7pb/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7719250Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.7719620Z 2022-11-23T01:34:53.7719757Z Running tests... 2022-11-23T01:34:53.7720162Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7720828Z test_basic_nccl_ckpt_always (__main__.TensorPipePipeWithDDPTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-11-23T01:34:53.7721385Z 2022-11-23T01:34:53.7721599Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7721839Z Ran 1 test in 0.000s 2022-11-23T01:34:53.7721951Z 2022-11-23T01:34:53.7722009Z OK (skipped=1) 2022-11-23T01:34:53.7722115Z 2022-11-23T01:34:53.7722198Z Generating XML reports... 2022-11-23T01:34:53.7722794Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123012846.xml 2022-11-23T01:34:53.7723675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7724167Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7724832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7725369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7725893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6cn0q3wh 2022-11-23T01:34:53.7726502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6cn0q3wh/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7727251Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.7727580Z 2022-11-23T01:34:53.7727693Z Running tests... 2022-11-23T01:34:53.7728047Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7728425Z test_basic_nccl_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-11-23T01:34:53.7728658Z 2022-11-23T01:34:53.7728852Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7729085Z Ran 1 test in 0.000s 2022-11-23T01:34:53.7729197Z 2022-11-23T01:34:53.7729255Z OK (skipped=1) 2022-11-23T01:34:53.7729359Z 2022-11-23T01:34:53.7729444Z Generating XML reports... 2022-11-23T01:34:53.7729940Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123012848.xml 2022-11-23T01:34:53.7730504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7730822Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7731250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7731597Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7731930Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8rukneta 2022-11-23T01:34:53.7732335Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8rukneta/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7732821Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.7733040Z 2022-11-23T01:34:53.7733111Z Running tests... 2022-11-23T01:34:53.7733398Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7733769Z test_basic_nccl_ckpt_never (__main__.TensorPipePipeWithDDPTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-11-23T01:34:53.7734070Z 2022-11-23T01:34:53.7734268Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7734495Z Ran 1 test in 0.000s 2022-11-23T01:34:53.7734607Z 2022-11-23T01:34:53.7734678Z OK (skipped=1) 2022-11-23T01:34:53.7734784Z 2022-11-23T01:34:53.7734867Z Generating XML reports... 2022-11-23T01:34:53.7735365Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123012850.xml 2022-11-23T01:34:53.7735917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7736290Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7736718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7737062Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7737396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaplgzwuv 2022-11-23T01:34:53.7737797Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaplgzwuv/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7738278Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.7738494Z 2022-11-23T01:34:53.7738554Z Running tests... 2022-11-23T01:34:53.7738854Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7739238Z test_basic_nccl_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) ... skip: c10d was not compiled with the NCCL backend (0.000s) 2022-11-23T01:34:53.7739474Z 2022-11-23T01:34:53.7739667Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7739889Z Ran 1 test in 0.000s 2022-11-23T01:34:53.7740001Z 2022-11-23T01:34:53.7740071Z OK (skipped=1) 2022-11-23T01:34:53.7740178Z 2022-11-23T01:34:53.7740263Z Generating XML reports... 2022-11-23T01:34:53.7740742Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123012851.xml 2022-11-23T01:34:53.7741306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7741634Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7742059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7742394Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7742741Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgxe8vmij 2022-11-23T01:34:53.7743145Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgxe8vmij/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7743632Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.7743837Z 2022-11-23T01:34:53.7743937Z Running tests... 2022-11-23T01:34:53.7744264Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7744722Z test_async_execution_nested_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29954 2022-11-23T01:34:53.7745197Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29955 2022-11-23T01:34:53.7745535Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29956 2022-11-23T01:34:53.7745870Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29957 2022-11-23T01:34:53.7746329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7746647Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7747129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7747486Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7747837Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw30449kv 2022-11-23T01:34:53.7748223Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw30449kv/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7748719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7749051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7749500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7749849Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7750198Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9belsnl_ 2022-11-23T01:34:53.7750597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9belsnl_/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7751072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7751399Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7751826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7752165Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7752590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsnlk5p3h 2022-11-23T01:34:53.7753033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsnlk5p3h/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7753531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7753869Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7754284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7754637Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7754985Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu8344jj3 2022-11-23T01:34:53.7755369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu8344jj3/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7755758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.7756121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.7756477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.7756827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.7757103Z skip: Need at least 1 CUDA device (2.179s) 2022-11-23T01:34:53.7757240Z 2022-11-23T01:34:53.7757439Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7757682Z Ran 1 test in 2.179s 2022-11-23T01:34:53.7757794Z 2022-11-23T01:34:53.7757852Z OK (skipped=1) 2022-11-23T01:34:53.7757961Z 2022-11-23T01:34:53.7758045Z Generating XML reports... 2022-11-23T01:34:53.7758585Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012853.xml 2022-11-23T01:34:53.7759196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7759512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7759976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7760326Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7760822Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjqe98xoc 2022-11-23T01:34:53.7761236Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjqe98xoc/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7761726Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.7761947Z 2022-11-23T01:34:53.7762022Z Running tests... 2022-11-23T01:34:53.7762314Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7762819Z test_async_execution_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30009 2022-11-23T01:34:53.7763267Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30010 2022-11-23T01:34:53.7763607Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30011 2022-11-23T01:34:53.7763960Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30012 2022-11-23T01:34:53.7764419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7764749Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7765172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7765531Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7765885Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp76edell5 2022-11-23T01:34:53.7766286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp76edell5/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7766763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7767093Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7767522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7767865Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7768196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7dg9slm5 2022-11-23T01:34:53.7768598Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7dg9slm5/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7769089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7769405Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7769834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7770180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7770523Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvwp84yq6 2022-11-23T01:34:53.7770912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvwp84yq6/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7771406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7771742Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7772164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7772530Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7772891Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgj4mj2cm 2022-11-23T01:34:53.7773341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgj4mj2cm/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7773722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.7774058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.7774399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.7774745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.7775018Z skip: Need at least 1 CUDA device (2.179s) 2022-11-23T01:34:53.7775154Z 2022-11-23T01:34:53.7775387Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7775631Z Ran 1 test in 2.179s 2022-11-23T01:34:53.7775741Z 2022-11-23T01:34:53.7775814Z OK (skipped=1) 2022-11-23T01:34:53.7775906Z 2022-11-23T01:34:53.7775991Z Generating XML reports... 2022-11-23T01:34:53.7776530Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012857.xml 2022-11-23T01:34:53.7777130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7777453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7777891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7778238Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7778603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvgqo2mph 2022-11-23T01:34:53.7779014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvgqo2mph/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7779508Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.7779750Z 2022-11-23T01:34:53.7779850Z Running tests... 2022-11-23T01:34:53.7780200Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7780653Z test_cuda_future_callback_changes_devices (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30064 2022-11-23T01:34:53.7781089Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30065 2022-11-23T01:34:53.7781418Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30066 2022-11-23T01:34:53.7781747Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30067 2022-11-23T01:34:53.7782191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7782542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7782980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7783330Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7783663Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2g3urifj 2022-11-23T01:34:53.7784074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2g3urifj/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7784717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7785332Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7785771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7786113Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7786456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2sinrza1 2022-11-23T01:34:53.7786900Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2sinrza1/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7787373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7787703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7788129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7788462Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7788804Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuc1s_3ah 2022-11-23T01:34:53.7789239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuc1s_3ah/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7789724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7790041Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7790468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7790821Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7791163Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9yi4upse 2022-11-23T01:34:53.7791550Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9yi4upse/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7791926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.7792277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.7792681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.7793019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.7793307Z skip: Need at least 2 CUDA devices (2.184s) 2022-11-23T01:34:53.7793444Z 2022-11-23T01:34:53.7793646Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7793873Z Ran 1 test in 2.185s 2022-11-23T01:34:53.7793985Z 2022-11-23T01:34:53.7794057Z OK (skipped=1) 2022-11-23T01:34:53.7794163Z 2022-11-23T01:34:53.7794247Z Generating XML reports... 2022-11-23T01:34:53.7794811Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012901.xml 2022-11-23T01:34:53.7795717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7796232Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7796859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7797485Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7797897Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpssus1gyq 2022-11-23T01:34:53.7798304Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpssus1gyq/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7798780Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.7798998Z 2022-11-23T01:34:53.7799072Z Running tests... 2022-11-23T01:34:53.7799375Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7799839Z test_cuda_future_can_extract_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30119 2022-11-23T01:34:53.7800278Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30120 2022-11-23T01:34:53.7800833Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30121 2022-11-23T01:34:53.7801174Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30122 2022-11-23T01:34:53.7801633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7801951Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7802381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7802727Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7803104Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4jvy3z45 2022-11-23T01:34:53.7803504Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4jvy3z45/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7803997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7804324Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7804737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7805081Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7805422Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbfevj7df 2022-11-23T01:34:53.7805900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7806216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7806642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7806985Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7807320Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjzspydxs 2022-11-23T01:34:53.7807818Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbfevj7df/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7808428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjzspydxs/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7809222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7809684Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7810468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7810824Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7811156Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1vh7ex_v 2022-11-23T01:34:53.7811563Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1vh7ex_v/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7811941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.7812290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.7812623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.7812960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.7813245Z skip: Need at least 1 CUDA device (2.281s) 2022-11-23T01:34:53.7813380Z 2022-11-23T01:34:53.7813581Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7813809Z Ran 1 test in 2.281s 2022-11-23T01:34:53.7813921Z 2022-11-23T01:34:53.7813994Z OK (skipped=1) 2022-11-23T01:34:53.7814102Z 2022-11-23T01:34:53.7814187Z Generating XML reports... 2022-11-23T01:34:53.7814785Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012905.xml 2022-11-23T01:34:53.7815392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7815725Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7816154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7816491Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7816838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnww5zahb 2022-11-23T01:34:53.7817280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnww5zahb/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7817751Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.7817969Z 2022-11-23T01:34:53.7818044Z Running tests... 2022-11-23T01:34:53.7818348Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7818794Z test_cuda_future_can_extract_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30174 2022-11-23T01:34:53.7819226Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30175 2022-11-23T01:34:53.7819558Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30176 2022-11-23T01:34:53.7819882Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30177 2022-11-23T01:34:53.7820344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7820663Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7821092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7821439Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7821771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp99jfklr 2022-11-23T01:34:53.7822173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp99jfklr/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7822656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7822983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7823400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7823749Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7824096Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptlo8rkh2 2022-11-23T01:34:53.7824486Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptlo8rkh2/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7824975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7825305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7825732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7826064Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7826405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzih9v67f 2022-11-23T01:34:53.7826814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzih9v67f/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7827294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7827640Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7828067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7828410Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7828741Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp665lbiod 2022-11-23T01:34:53.7829141Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp665lbiod/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7829518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.7829899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.7830229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.7830568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.7830854Z skip: Need at least 1 CUDA device (2.295s) 2022-11-23T01:34:53.7830988Z 2022-11-23T01:34:53.7831176Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7831433Z Ran 1 test in 2.295s 2022-11-23T01:34:53.7831596Z 2022-11-23T01:34:53.7831702Z OK (skipped=1) 2022-11-23T01:34:53.7831852Z 2022-11-23T01:34:53.7831972Z Generating XML reports... 2022-11-23T01:34:53.7832948Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012909.xml 2022-11-23T01:34:53.7833939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7834506Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7835246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7835867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7836492Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc74prjyv 2022-11-23T01:34:53.7837225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc74prjyv/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7838100Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.7838499Z 2022-11-23T01:34:53.7838624Z Running tests... 2022-11-23T01:34:53.7839158Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7840005Z test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30229 2022-11-23T01:34:53.7840950Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30230 2022-11-23T01:34:53.7841550Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30231 2022-11-23T01:34:53.7842148Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30232 2022-11-23T01:34:53.7842961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7843549Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7844329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7844951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7845555Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx77cbke_ 2022-11-23T01:34:53.7846297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx77cbke_/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7847315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7847913Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7848684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7849309Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7849929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph3u3sd44 2022-11-23T01:34:53.7850643Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph3u3sd44/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7851527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7852185Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7852968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7853582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7854197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8kvls390 2022-11-23T01:34:53.7854907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8kvls390/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7855779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7856369Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7857158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7857785Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7858392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc4ru27si 2022-11-23T01:34:53.7859126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc4ru27si/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7859808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.7860443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.7861060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.7861683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.7862192Z skip: Need at least 1 CUDA device (2.282s) 2022-11-23T01:34:53.7862430Z 2022-11-23T01:34:53.7862773Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7863194Z Ran 1 test in 2.282s 2022-11-23T01:34:53.7863390Z 2022-11-23T01:34:53.7863513Z OK (skipped=1) 2022-11-23T01:34:53.7863700Z 2022-11-23T01:34:53.7863850Z Generating XML reports... 2022-11-23T01:34:53.7864839Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012913.xml 2022-11-23T01:34:53.7865941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7866531Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7867303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7867922Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7868541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw35oja8k 2022-11-23T01:34:53.7869274Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw35oja8k/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7870136Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.7870532Z 2022-11-23T01:34:53.7870655Z Running tests... 2022-11-23T01:34:53.7871229Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7872059Z test_cuda_future_can_extract_custom_class_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30284 2022-11-23T01:34:53.7872954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30285 2022-11-23T01:34:53.7873558Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30286 2022-11-23T01:34:53.7874150Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30287 2022-11-23T01:34:53.7875003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7875575Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7876331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7876951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7877560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpic3b0lh3 2022-11-23T01:34:53.7878290Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpic3b0lh3/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7879197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7879798Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7880549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7881282Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7881911Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp57wannvw 2022-11-23T01:34:53.7882643Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp57wannvw/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7883538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7884128Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7884910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7885516Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7886139Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsq8wnovw 2022-11-23T01:34:53.7886883Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsq8wnovw/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7887763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7888358Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7889137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7889759Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7890356Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsp8hwsqr 2022-11-23T01:34:53.7891089Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsp8hwsqr/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7891781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.7892421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.7893039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.7893663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.7894245Z skip: Need at least 1 CUDA device (2.187s) 2022-11-23T01:34:53.7894486Z 2022-11-23T01:34:53.7894828Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7895247Z Ran 1 test in 2.187s 2022-11-23T01:34:53.7895443Z 2022-11-23T01:34:53.7895566Z OK (skipped=1) 2022-11-23T01:34:53.7895748Z 2022-11-23T01:34:53.7895891Z Generating XML reports... 2022-11-23T01:34:53.7896857Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012917.xml 2022-11-23T01:34:53.7897959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7898594Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7899367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7899994Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7900612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp17tj8bx7 2022-11-23T01:34:53.7901340Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp17tj8bx7/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7902214Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.7902607Z 2022-11-23T01:34:53.7902734Z Running tests... 2022-11-23T01:34:53.7903267Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7904088Z test_cuda_future_can_extract_list_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30339 2022-11-23T01:34:53.7904927Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30340 2022-11-23T01:34:53.7905526Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30341 2022-11-23T01:34:53.7906129Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30342 2022-11-23T01:34:53.7906935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7907522Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7908298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7908916Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7909526Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvjs60a37 2022-11-23T01:34:53.7910252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvjs60a37/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7911115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7911607Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7912471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7912882Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7913228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwsj71blu 2022-11-23T01:34:53.7913619Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwsj71blu/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7914113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7914446Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7914874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7915266Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7915613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjwhgi1ax 2022-11-23T01:34:53.7916012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjwhgi1ax/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7916485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7916809Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7917237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7917609Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7917940Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptgf5fqor 2022-11-23T01:34:53.7918347Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptgf5fqor/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7918729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.7919064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.7919408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.7919752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.7920039Z skip: Need at least 1 CUDA device (2.282s) 2022-11-23T01:34:53.7920159Z 2022-11-23T01:34:53.7920364Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7920750Z Ran 1 test in 2.282s 2022-11-23T01:34:53.7920876Z 2022-11-23T01:34:53.7920950Z OK (skipped=1) 2022-11-23T01:34:53.7921057Z 2022-11-23T01:34:53.7921128Z Generating XML reports... 2022-11-23T01:34:53.7921671Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012921.xml 2022-11-23T01:34:53.7922278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7922609Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7923023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7923373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7923722Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpajd0k7zs 2022-11-23T01:34:53.7924125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpajd0k7zs/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7924593Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.7924812Z 2022-11-23T01:34:53.7924888Z Running tests... 2022-11-23T01:34:53.7925193Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7925635Z test_cuda_future_can_extract_list_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30394 2022-11-23T01:34:53.7926092Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30395 2022-11-23T01:34:53.7926423Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30396 2022-11-23T01:34:53.7926748Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30397 2022-11-23T01:34:53.7927190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7927533Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7927967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7928373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7928706Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb5kozddh 2022-11-23T01:34:53.7929110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb5kozddh/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7929598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7929910Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7930337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7930719Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7931060Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu_ylr4lh 2022-11-23T01:34:53.7931448Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu_ylr4lh/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7931936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7932264Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7932694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7933025Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7933365Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbmin09z0 2022-11-23T01:34:53.7933766Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbmin09z0/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7934293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7934797Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7935447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7936011Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7936561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp801ivvu9 2022-11-23T01:34:53.7937208Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp801ivvu9/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7937891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.7938501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.7939127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.7939751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.7940263Z skip: Need at least 1 CUDA device (2.179s) 2022-11-23T01:34:53.7940483Z 2022-11-23T01:34:53.7940861Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7941289Z Ran 1 test in 2.180s 2022-11-23T01:34:53.7941488Z 2022-11-23T01:34:53.7941610Z OK (skipped=1) 2022-11-23T01:34:53.7941794Z 2022-11-23T01:34:53.7941924Z Generating XML reports... 2022-11-23T01:34:53.7942925Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012926.xml 2022-11-23T01:34:53.7944054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7944652Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7945444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7946148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7946775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjxo3w8j6 2022-11-23T01:34:53.7947520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjxo3w8j6/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7948408Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.7948807Z 2022-11-23T01:34:53.7948937Z Running tests... 2022-11-23T01:34:53.7949479Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7950257Z test_cuda_future_device_as_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30449 2022-11-23T01:34:53.7951115Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30450 2022-11-23T01:34:53.7951723Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30451 2022-11-23T01:34:53.7952424Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30452 2022-11-23T01:34:53.7953251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7953856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7954648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7955274Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7955890Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt57fqmtp 2022-11-23T01:34:53.7956639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt57fqmtp/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7957559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7958137Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7958934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7959562Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7960195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphw8nvcxf 2022-11-23T01:34:53.7961101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphw8nvcxf/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7962022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7962631Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7963431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7964048Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7964665Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp91l3jeqa 2022-11-23T01:34:53.7965399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp91l3jeqa/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7966287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7966893Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7967686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7968307Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7968908Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpko8xzr6u 2022-11-23T01:34:53.7969646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpko8xzr6u/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7970436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.7971070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.7971689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.7972314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.7972827Z skip: Need at least 1 CUDA device (2.182s) 2022-11-23T01:34:53.7973063Z 2022-11-23T01:34:53.7973413Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7973910Z Ran 1 test in 2.182s 2022-11-23T01:34:53.7974106Z 2022-11-23T01:34:53.7974227Z OK (skipped=1) 2022-11-23T01:34:53.7974415Z 2022-11-23T01:34:53.7974565Z Generating XML reports... 2022-11-23T01:34:53.7975573Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012930.xml 2022-11-23T01:34:53.7976708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7977308Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7978102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7978716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7979336Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpms05h9dg 2022-11-23T01:34:53.7980076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpms05h9dg/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7980963Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.7981365Z 2022-11-23T01:34:53.7981492Z Running tests... 2022-11-23T01:34:53.7982057Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.7982852Z test_cuda_future_device_as_int (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30504 2022-11-23T01:34:53.7983647Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30505 2022-11-23T01:34:53.7984255Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30506 2022-11-23T01:34:53.7984850Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30507 2022-11-23T01:34:53.7985686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7986275Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7987073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7987700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7988295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7jirfkwl 2022-11-23T01:34:53.7989039Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7jirfkwl/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7989955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7990560Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7991330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7991968Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7992673Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmposon_pcs 2022-11-23T01:34:53.7993526Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmposon_pcs/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7994425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7995009Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.7995796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.7996411Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.7997041Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9rxeqyi1 2022-11-23T01:34:53.7997769Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9rxeqyi1/_remote_module_non_scriptable.py 2022-11-23T01:34:53.7998728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.7999317Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8000123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8000968Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8001601Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphv6uob0r 2022-11-23T01:34:53.8002328Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphv6uob0r/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8003024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8003658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8004279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8004910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8005423Z skip: Need at least 1 CUDA device (2.276s) 2022-11-23T01:34:53.8005666Z 2022-11-23T01:34:53.8006029Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8006439Z Ran 1 test in 2.277s 2022-11-23T01:34:53.8006637Z 2022-11-23T01:34:53.8006762Z OK (skipped=1) 2022-11-23T01:34:53.8006949Z 2022-11-23T01:34:53.8007099Z Generating XML reports... 2022-11-23T01:34:53.8008091Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012934.xml 2022-11-23T01:34:53.8009207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8009811Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8010595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8011212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8011843Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu24_5gy4 2022-11-23T01:34:53.8012569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu24_5gy4/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8013448Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8013822Z 2022-11-23T01:34:53.8013955Z Running tests... 2022-11-23T01:34:53.8014495Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8015281Z test_cuda_future_device_as_str (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30559 2022-11-23T01:34:53.8016083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30560 2022-11-23T01:34:53.8016702Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30561 2022-11-23T01:34:53.8017410Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30562 2022-11-23T01:34:53.8018267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8018837Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8019630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8020267Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8020889Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpww8ftjt5 2022-11-23T01:34:53.8021604Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpww8ftjt5/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8022611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8023216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8023998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8024628Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8025234Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprfg2se71 2022-11-23T01:34:53.8025971Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprfg2se71/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8026876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8027476Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8028283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8028856Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8029370Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplbwrp2w_ 2022-11-23T01:34:53.8030100Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplbwrp2w_/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8031004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8031589Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8032456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8033095Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8033716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb1soa8li 2022-11-23T01:34:53.8034430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb1soa8li/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8035136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8035780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8036425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8037042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8037552Z skip: Need at least 1 CUDA device (2.279s) 2022-11-23T01:34:53.8037790Z 2022-11-23T01:34:53.8038168Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8038576Z Ran 1 test in 2.279s 2022-11-23T01:34:53.8038776Z 2022-11-23T01:34:53.8038897Z OK (skipped=1) 2022-11-23T01:34:53.8039085Z 2022-11-23T01:34:53.8039237Z Generating XML reports... 2022-11-23T01:34:53.8040253Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012938.xml 2022-11-23T01:34:53.8041620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8042226Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8043020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8043653Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8044277Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwh8b7j1z 2022-11-23T01:34:53.8045021Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwh8b7j1z/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8046005Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8046408Z 2022-11-23T01:34:53.8046526Z Running tests... 2022-11-23T01:34:53.8047065Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8047876Z test_cuda_future_device_not_cuda (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30614 2022-11-23T01:34:53.8048690Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30615 2022-11-23T01:34:53.8049253Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30616 2022-11-23T01:34:53.8049855Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30617 2022-11-23T01:34:53.8050705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8051292Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8052089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8052724Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8053357Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq17782b1 2022-11-23T01:34:53.8054085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq17782b1/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8054982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8055588Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8056382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8056986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8057609Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnj_tajkp 2022-11-23T01:34:53.8058341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnj_tajkp/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8059241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8059843Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8060653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8061278Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8061886Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4h9vocf7 2022-11-23T01:34:53.8062612Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4h9vocf7/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8063521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8064124Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8064967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8065577Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8066214Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpev9j8gtf 2022-11-23T01:34:53.8066940Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpev9j8gtf/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8067626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8068267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8068897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8069571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8070083Z skip: Need at least 1 CUDA device (2.281s) 2022-11-23T01:34:53.8070325Z 2022-11-23T01:34:53.8070704Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8071117Z Ran 1 test in 2.281s 2022-11-23T01:34:53.8071303Z 2022-11-23T01:34:53.8071428Z OK (skipped=1) 2022-11-23T01:34:53.8071618Z 2022-11-23T01:34:53.8071763Z Generating XML reports... 2022-11-23T01:34:53.8072851Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012942.xml 2022-11-23T01:34:53.8073972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8074571Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8075377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8076005Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8076623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpduljjs9b 2022-11-23T01:34:53.8077371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpduljjs9b/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8078274Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8078668Z 2022-11-23T01:34:53.8078795Z Running tests... 2022-11-23T01:34:53.8079316Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8080134Z test_cuda_future_modify_tensor_inplace (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30669 2022-11-23T01:34:53.8081092Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30670 2022-11-23T01:34:53.8081687Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30671 2022-11-23T01:34:53.8110196Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30672 2022-11-23T01:34:53.8111079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8111685Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8112551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8113158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8113782Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsc0wprai 2022-11-23T01:34:53.8114519Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsc0wprai/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8115428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8116024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8116960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8117582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8118186Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0g78t1i1 2022-11-23T01:34:53.8118916Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0g78t1i1/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8119818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8120408Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8121499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8122130Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8122763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo0kd6dc4 2022-11-23T01:34:53.8123495Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo0kd6dc4/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8124409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8125012Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8125805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8126421Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8127039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwbjy3vdx 2022-11-23T01:34:53.8127786Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwbjy3vdx/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8128489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8129108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8129740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8130376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8130869Z skip: Need at least 1 CUDA device (2.279s) 2022-11-23T01:34:53.8131110Z 2022-11-23T01:34:53.8131468Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8131898Z Ran 1 test in 2.279s 2022-11-23T01:34:53.8132095Z 2022-11-23T01:34:53.8132219Z OK (skipped=1) 2022-11-23T01:34:53.8132398Z 2022-11-23T01:34:53.8132545Z Generating XML reports... 2022-11-23T01:34:53.8133563Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012946.xml 2022-11-23T01:34:53.8134697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8135283Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8136071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8136700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8137334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp928ut79g 2022-11-23T01:34:53.8138059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp928ut79g/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8138948Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8139338Z 2022-11-23T01:34:53.8139468Z Running tests... 2022-11-23T01:34:53.8140014Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8140946Z test_cuda_future_replace_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30724 2022-11-23T01:34:53.8141760Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30725 2022-11-23T01:34:53.8142380Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30726 2022-11-23T01:34:53.8142964Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30727 2022-11-23T01:34:53.8143818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8144410Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8145292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8145899Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8146541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpff__wt9p 2022-11-23T01:34:53.8147275Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpff__wt9p/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8148198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8148789Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8149574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8150208Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8150829Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdipdusvj 2022-11-23T01:34:53.8151574Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdipdusvj/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8152564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8153163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8153949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8154582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8155222Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwmnggib_ 2022-11-23T01:34:53.8155960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwmnggib_/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8156838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8157443Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8158239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8158852Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8159483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbxi4eojg 2022-11-23T01:34:53.8160228Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbxi4eojg/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8161043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8161601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8162245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8162893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8163403Z skip: Need at least 1 CUDA device (2.179s) 2022-11-23T01:34:53.8163638Z 2022-11-23T01:34:53.8164018Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8164555Z Ran 1 test in 2.179s 2022-11-23T01:34:53.8164762Z 2022-11-23T01:34:53.8164888Z OK (skipped=1) 2022-11-23T01:34:53.8165065Z 2022-11-23T01:34:53.8165210Z Generating XML reports... 2022-11-23T01:34:53.8166231Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012950.xml 2022-11-23T01:34:53.8167353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8167969Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8168759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8169447Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8170076Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxrgxsa0v 2022-11-23T01:34:53.8170804Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxrgxsa0v/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8171693Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8172083Z 2022-11-23T01:34:53.8172208Z Running tests... 2022-11-23T01:34:53.8172755Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8173554Z test_cuda_future_value_on_bad_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30779 2022-11-23T01:34:53.8174372Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30780 2022-11-23T01:34:53.8174991Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30781 2022-11-23T01:34:53.8175588Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30782 2022-11-23T01:34:53.8176445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8177051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8177845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8178471Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8179108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2a8ae6ov 2022-11-23T01:34:53.8179847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2a8ae6ov/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8180753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8181331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8182091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8182654Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8183179Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_0ruldym 2022-11-23T01:34:53.8183775Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_0ruldym/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8184668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8185259Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8186025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8186652Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8187281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd4wt881k 2022-11-23T01:34:53.8188066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd4wt881k/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8188852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8189405Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8190146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8190723Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8191274Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgci0uvau 2022-11-23T01:34:53.8191917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgci0uvau/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8192673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8193133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8193476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8193819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8194089Z skip: Need at least 2 CUDA devices (2.280s) 2022-11-23T01:34:53.8194229Z 2022-11-23T01:34:53.8194447Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8194689Z Ran 1 test in 2.281s 2022-11-23T01:34:53.8194804Z 2022-11-23T01:34:53.8194881Z OK (skipped=1) 2022-11-23T01:34:53.8194976Z 2022-11-23T01:34:53.8195062Z Generating XML reports... 2022-11-23T01:34:53.8195609Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012954.xml 2022-11-23T01:34:53.8196207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8196530Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8196961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8197303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8197647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbb42zqpg 2022-11-23T01:34:53.8198036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbb42zqpg/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8198519Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8198737Z 2022-11-23T01:34:53.8198809Z Running tests... 2022-11-23T01:34:53.8199109Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8199989Z test_custom_stream (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/79750 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.255s) 2022-11-23T01:34:53.8200545Z 2022-11-23T01:34:53.8200897Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8201140Z Ran 1 test in 0.255s 2022-11-23T01:34:53.8201254Z 2022-11-23T01:34:53.8201324Z OK (skipped=1) 2022-11-23T01:34:53.8201430Z 2022-11-23T01:34:53.8201513Z Generating XML reports... 2022-11-23T01:34:53.8202035Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012958.xml 2022-11-23T01:34:53.8202636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8203063Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8203488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8203837Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8204187Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkt_u0rtp 2022-11-23T01:34:53.8204591Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkt_u0rtp/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8205063Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8205323Z 2022-11-23T01:34:53.8205395Z Running tests... 2022-11-23T01:34:53.8205698Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8206115Z test_custom_stream_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30844 2022-11-23T01:34:53.8206546Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30845 2022-11-23T01:34:53.8206882Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30846 2022-11-23T01:34:53.8207208Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30847 2022-11-23T01:34:53.8207647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8207978Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8208410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8208759Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8209090Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpseeskcs6 2022-11-23T01:34:53.8209496Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpseeskcs6/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8209986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8210304Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8210733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8211078Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8211419Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw7jiwrsn 2022-11-23T01:34:53.8211812Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw7jiwrsn/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8212299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8212628Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8213042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8213386Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8213731Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp27h89zfv 2022-11-23T01:34:53.8214127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp27h89zfv/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8214596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8214923Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8215354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8215698Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8216091Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp07qfmxc6 2022-11-23T01:34:53.8216493Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp07qfmxc6/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8216871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8217205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8217550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8217898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8218220Z skip: Need at least 2 CUDA devices (2.280s) 2022-11-23T01:34:53.8218343Z 2022-11-23T01:34:53.8218544Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8218787Z Ran 1 test in 2.280s 2022-11-23T01:34:53.8218900Z 2022-11-23T01:34:53.8218974Z OK (skipped=1) 2022-11-23T01:34:53.8219083Z 2022-11-23T01:34:53.8219154Z Generating XML reports... 2022-11-23T01:34:53.8219693Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013000.xml 2022-11-23T01:34:53.8220297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8220627Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8221039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8221386Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8221730Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpomt4qy01 2022-11-23T01:34:53.8222134Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpomt4qy01/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8222602Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8222821Z 2022-11-23T01:34:53.8222892Z Running tests... 2022-11-23T01:34:53.8223194Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8223607Z test_custom_stream_nested (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30899 2022-11-23T01:34:53.8224037Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30900 2022-11-23T01:34:53.8224367Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30901 2022-11-23T01:34:53.8224698Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30902 2022-11-23T01:34:53.8225136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8225465Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8225897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8226234Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8226577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfl83va5w 2022-11-23T01:34:53.8226980Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfl83va5w/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8227465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8227782Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8228209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8228560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8228948Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsgu6erby 2022-11-23T01:34:53.8229341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsgu6erby/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8229829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8230163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8230576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8230925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8231301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk2ix3phw 2022-11-23T01:34:53.8231707Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk2ix3phw/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8232182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8232606Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8233038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8233373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8233723Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgcmgl70_ 2022-11-23T01:34:53.8234134Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgcmgl70_/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8234515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8234858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8235204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8235555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8235842Z skip: Need at least 2 CUDA devices (2.282s) 2022-11-23T01:34:53.8235965Z 2022-11-23T01:34:53.8236167Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8236407Z Ran 1 test in 2.282s 2022-11-23T01:34:53.8236524Z 2022-11-23T01:34:53.8236597Z OK (skipped=1) 2022-11-23T01:34:53.8236703Z 2022-11-23T01:34:53.8236774Z Generating XML reports... 2022-11-23T01:34:53.8237311Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013004.xml 2022-11-23T01:34:53.8237918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8238250Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8238669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8239011Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8239358Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnnev_7dg 2022-11-23T01:34:53.8239748Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnnev_7dg/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8240229Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8240446Z 2022-11-23T01:34:53.8240519Z Running tests... 2022-11-23T01:34:53.8240984Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8241409Z test_custom_stream_nested_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30954 2022-11-23T01:34:53.8241911Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30955 2022-11-23T01:34:53.8242247Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30956 2022-11-23T01:34:53.8242576Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30957 2022-11-23T01:34:53.8243015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8243344Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8243775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8244150Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8244493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2n9kn8f4 2022-11-23T01:34:53.8244895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2n9kn8f4/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8245387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8245699Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8246124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8246466Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8246789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpelsmofrs 2022-11-23T01:34:53.8247184Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpelsmofrs/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8247671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8247997Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8248407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8248749Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8249085Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmnh03wac 2022-11-23T01:34:53.8249479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmnh03wac/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8249951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8250276Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8250704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8251036Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8251370Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0l3naryx 2022-11-23T01:34:53.8251564Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0l3naryx/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8251727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8251889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8252048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8252205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8252300Z skip: Need at least 2 CUDA devices (2.292s) 2022-11-23T01:34:53.8252308Z 2022-11-23T01:34:53.8252503Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8252576Z Ran 1 test in 2.292s 2022-11-23T01:34:53.8252581Z 2022-11-23T01:34:53.8252652Z OK (skipped=1) 2022-11-23T01:34:53.8252656Z 2022-11-23T01:34:53.8252739Z Generating XML reports... 2022-11-23T01:34:53.8253207Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013009.xml 2022-11-23T01:34:53.8253486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8253608Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8253882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8254004Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8254219Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0nq8mfkb 2022-11-23T01:34:53.8254412Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0nq8mfkb/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8254676Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8254682Z 2022-11-23T01:34:53.8254754Z Running tests... 2022-11-23T01:34:53.8254946Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8255209Z test_device_map_cpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31009 2022-11-23T01:34:53.8255364Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31010 2022-11-23T01:34:53.8255504Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31011 2022-11-23T01:34:53.8255659Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31012 2022-11-23T01:34:53.8255930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8256052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8256330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8256463Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8256648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvlz5hbw9 2022-11-23T01:34:53.8256844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvlz5hbw9/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8257118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8257227Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8257504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8257636Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8257819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvnavdjpl 2022-11-23T01:34:53.8258015Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvnavdjpl/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8258281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8258400Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8258680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8258814Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8258985Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmparesynv1 2022-11-23T01:34:53.8259180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmparesynv1/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8259485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8259605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8259881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8260015Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8260197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdbxehk0x 2022-11-23T01:34:53.8260390Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdbxehk0x/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8260541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8260731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8260891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8261053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8261123Z ok (2.582s) 2022-11-23T01:34:53.8261129Z 2022-11-23T01:34:53.8261324Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8261400Z Ran 1 test in 2.582s 2022-11-23T01:34:53.8261405Z 2022-11-23T01:34:53.8261463Z OK 2022-11-23T01:34:53.8261468Z 2022-11-23T01:34:53.8261538Z Generating XML reports... 2022-11-23T01:34:53.8261954Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013013.xml 2022-11-23T01:34:53.8262223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8262348Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8262625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8262761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8262942Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe6v1dgg4 2022-11-23T01:34:53.8263134Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe6v1dgg4/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8263401Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8263407Z 2022-11-23T01:34:53.8263467Z Running tests... 2022-11-23T01:34:53.8263660Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8263936Z test_device_map_cpu_to_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31232 2022-11-23T01:34:53.8264093Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31233 2022-11-23T01:34:53.8264246Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31234 2022-11-23T01:34:53.8264395Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31235 2022-11-23T01:34:53.8264669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8264788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8265072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8265194Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8265376Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo_cq6jp2 2022-11-23T01:34:53.8265572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo_cq6jp2/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8265841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8265998Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8266280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8266415Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8266599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe1xhq3xn 2022-11-23T01:34:53.8266782Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe1xhq3xn/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8267053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8267202Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8267484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8267619Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8267804Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfjvyy295 2022-11-23T01:34:53.8267996Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfjvyy295/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8268266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8268388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8268653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8268790Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8268973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdv9gin4v 2022-11-23T01:34:53.8269169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdv9gin4v/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8269329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8269493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8269657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8269815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8269916Z skip: Need at least 1 CUDA device (2.284s) 2022-11-23T01:34:53.8269922Z 2022-11-23T01:34:53.8270106Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8270182Z Ran 1 test in 2.284s 2022-11-23T01:34:53.8270186Z 2022-11-23T01:34:53.8270256Z OK (skipped=1) 2022-11-23T01:34:53.8270260Z 2022-11-23T01:34:53.8270339Z Generating XML reports... 2022-11-23T01:34:53.8270756Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013017.xml 2022-11-23T01:34:53.8271030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8271154Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8271437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8271560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8271745Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwmrke3_r 2022-11-23T01:34:53.8271944Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwmrke3_r/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8272210Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8272215Z 2022-11-23T01:34:53.8272285Z Running tests... 2022-11-23T01:34:53.8272628Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8272913Z test_device_map_cpu_to_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31287 2022-11-23T01:34:53.8273068Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31288 2022-11-23T01:34:53.8273221Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31289 2022-11-23T01:34:53.8273360Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31290 2022-11-23T01:34:53.8273634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8273789Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8274069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8274204Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8274390Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpch7n44oj 2022-11-23T01:34:53.8274585Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpch7n44oj/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8274855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8274980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8275245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8275381Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8275564Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbpdvs7u5 2022-11-23T01:34:53.8275762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbpdvs7u5/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8276028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8276151Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8276429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8276563Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8276734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp864o712w 2022-11-23T01:34:53.8276930Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp864o712w/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8277199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8277325Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8277601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8277734Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8277918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4xm879l2 2022-11-23T01:34:53.8278111Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4xm879l2/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8278272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8278421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8278585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8278746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8278879Z skip: Need at least 2 CUDA devices (2.281s) 2022-11-23T01:34:53.8278885Z 2022-11-23T01:34:53.8279082Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8279158Z Ran 1 test in 2.282s 2022-11-23T01:34:53.8279163Z 2022-11-23T01:34:53.8279235Z OK (skipped=1) 2022-11-23T01:34:53.8279239Z 2022-11-23T01:34:53.8279323Z Generating XML reports... 2022-11-23T01:34:53.8279731Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013021.xml 2022-11-23T01:34:53.8280001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8280151Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8280425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8280687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8280900Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc21clod5 2022-11-23T01:34:53.8281096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc21clod5/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8281368Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8281372Z 2022-11-23T01:34:53.8281444Z Running tests... 2022-11-23T01:34:53.8281625Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8281895Z test_device_map_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31342 2022-11-23T01:34:53.8282053Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31343 2022-11-23T01:34:53.8282205Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31344 2022-11-23T01:34:53.8282356Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31345 2022-11-23T01:34:53.8282629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8282751Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8283036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8283168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8283341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph1u1iv9d 2022-11-23T01:34:53.8283538Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph1u1iv9d/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8283806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8283929Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8284208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8284338Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8284523Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp65b_llj 2022-11-23T01:34:53.8284794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8284902Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8285178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8285312Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8285493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp20qequne 2022-11-23T01:34:53.8285743Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp65b_llj/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8285937Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp20qequne/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8286211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8286334Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8286614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8286734Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8286956Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmm6v7_9x 2022-11-23T01:34:53.8287146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmm6v7_9x/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8287313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8287479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8287640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8287802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8287903Z skip: Need at least 2 CUDA devices (2.179s) 2022-11-23T01:34:53.8287908Z 2022-11-23T01:34:53.8288101Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8288167Z Ran 1 test in 2.179s 2022-11-23T01:34:53.8288172Z 2022-11-23T01:34:53.8288241Z OK (skipped=1) 2022-11-23T01:34:53.8288246Z 2022-11-23T01:34:53.8288327Z Generating XML reports... 2022-11-23T01:34:53.8288743Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013025.xml 2022-11-23T01:34:53.8289012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8289135Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8289412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8289548Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8289719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfxhwdxr0 2022-11-23T01:34:53.8289918Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfxhwdxr0/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8290189Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8290193Z 2022-11-23T01:34:53.8290266Z Running tests... 2022-11-23T01:34:53.8290464Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8291228Z test_device_map_gpu_default_to_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/80008 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.257s) 2022-11-23T01:34:53.8291235Z 2022-11-23T01:34:53.8291425Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8291494Z Ran 1 test in 0.257s 2022-11-23T01:34:53.8291501Z 2022-11-23T01:34:53.8291571Z OK (skipped=1) 2022-11-23T01:34:53.8291575Z 2022-11-23T01:34:53.8291655Z Generating XML reports... 2022-11-23T01:34:53.8292060Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013029.xml 2022-11-23T01:34:53.8292365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8292492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8292772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8292906Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8293089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv_dcy6ib 2022-11-23T01:34:53.8293283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv_dcy6ib/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8293580Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8293585Z 2022-11-23T01:34:53.8293657Z Running tests... 2022-11-23T01:34:53.8293838Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8294105Z test_device_map_gpu_mixed_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31407 2022-11-23T01:34:53.8294262Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31408 2022-11-23T01:34:53.8294415Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31409 2022-11-23T01:34:53.8294564Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31410 2022-11-23T01:34:53.8294837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8294963Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8295241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8295374Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8295550Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5but4qzi 2022-11-23T01:34:53.8295744Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5but4qzi/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8296013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8296132Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8296405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8296540Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8296728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0lh9dgo7 2022-11-23T01:34:53.8296919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0lh9dgo7/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8297181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8297305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8297581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8297713Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8297893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0vinl898 2022-11-23T01:34:53.8298085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0vinl898/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8298356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8298477Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8298798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8298923Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8299104Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2l2zko_o 2022-11-23T01:34:53.8299299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2l2zko_o/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8299462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8299620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8299778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8299967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8300073Z skip: Need at least 2 CUDA devices (2.281s) 2022-11-23T01:34:53.8300078Z 2022-11-23T01:34:53.8300265Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8300339Z Ran 1 test in 2.281s 2022-11-23T01:34:53.8300345Z 2022-11-23T01:34:53.8300416Z OK (skipped=1) 2022-11-23T01:34:53.8300421Z 2022-11-23T01:34:53.8300503Z Generating XML reports... 2022-11-23T01:34:53.8300924Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013031.xml 2022-11-23T01:34:53.8301199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8301321Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8301603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8301737Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8301911Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphd8c324e 2022-11-23T01:34:53.8302109Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphd8c324e/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8302373Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8302379Z 2022-11-23T01:34:53.8302451Z Running tests... 2022-11-23T01:34:53.8302644Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8302910Z test_device_map_gpu_mixed_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31462 2022-11-23T01:34:53.8303067Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31463 2022-11-23T01:34:53.8303219Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31464 2022-11-23T01:34:53.8303371Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31465 2022-11-23T01:34:53.8303632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8303752Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8304031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8304166Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8304351Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpps93838i 2022-11-23T01:34:53.8304544Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpps93838i/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8304814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8304934Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8305237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8305376Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8305558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphmycuwyy 2022-11-23T01:34:53.8305752Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphmycuwyy/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8306026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8306148Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8306464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8306597Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8306781Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptp1l1ck6 2022-11-23T01:34:53.8306966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptp1l1ck6/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8307236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8307357Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8307632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8307765Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8307951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppyiv8w1e 2022-11-23T01:34:53.8308146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppyiv8w1e/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8308308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8308459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8308619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8308780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8308884Z skip: Need at least 2 CUDA devices (2.183s) 2022-11-23T01:34:53.8308890Z 2022-11-23T01:34:53.8309085Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8309163Z Ran 1 test in 2.183s 2022-11-23T01:34:53.8309168Z 2022-11-23T01:34:53.8309239Z OK (skipped=1) 2022-11-23T01:34:53.8309246Z 2022-11-23T01:34:53.8309328Z Generating XML reports... 2022-11-23T01:34:53.8309747Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013036.xml 2022-11-23T01:34:53.8310011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8310137Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8310417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8310551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8310735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphxav909x 2022-11-23T01:34:53.8310933Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphxav909x/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8311200Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8311208Z 2022-11-23T01:34:53.8311277Z Running tests... 2022-11-23T01:34:53.8311473Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8311762Z test_device_map_gpu_mixed_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31517 2022-11-23T01:34:53.8311920Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31518 2022-11-23T01:34:53.8312073Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31519 2022-11-23T01:34:53.8312226Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31520 2022-11-23T01:34:53.8312594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8312719Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8313044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8313178Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8313352Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvuaj65nj 2022-11-23T01:34:53.8313549Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvuaj65nj/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8313818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8313944Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8314225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8314360Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8314547Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqw8_jwha 2022-11-23T01:34:53.8314742Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqw8_jwha/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8315017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8315128Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8315405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8315540Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8315722Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpotix0z78 2022-11-23T01:34:53.8315919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpotix0z78/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8316187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8316311Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8316586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8316720Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8316890Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1df6_ip_ 2022-11-23T01:34:53.8317084Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1df6_ip_/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8317246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8317407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8317565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8317728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8317832Z skip: Need at least 2 CUDA devices (2.180s) 2022-11-23T01:34:53.8317837Z 2022-11-23T01:34:53.8318032Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8318127Z Ran 1 test in 2.180s 2022-11-23T01:34:53.8318132Z 2022-11-23T01:34:53.8318204Z OK (skipped=1) 2022-11-23T01:34:53.8318208Z 2022-11-23T01:34:53.8318292Z Generating XML reports... 2022-11-23T01:34:53.8318708Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013040.xml 2022-11-23T01:34:53.8318978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8319101Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8319380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8319543Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8319728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaw0vrxui 2022-11-23T01:34:53.8319913Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaw0vrxui/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8320181Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8320186Z 2022-11-23T01:34:53.8320260Z Running tests... 2022-11-23T01:34:53.8320455Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8320871Z test_device_map_gpu_mixed_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31572 2022-11-23T01:34:53.8321031Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31573 2022-11-23T01:34:53.8321190Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31574 2022-11-23T01:34:53.8321343Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31575 2022-11-23T01:34:53.8321622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8321732Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8322013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8322146Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8322330Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxfy74t6n 2022-11-23T01:34:53.8322525Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxfy74t6n/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8322797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8322924Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8323204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8323327Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8323511Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp266499tq 2022-11-23T01:34:53.8323703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp266499tq/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8323975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8324095Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8324375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8324510Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8324729Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4gd_qha9 2022-11-23T01:34:53.8325164Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4gd_qha9/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8325609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8325736Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8326020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8326156Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8326344Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4z8auv0r 2022-11-23T01:34:53.8326610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4z8auv0r/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8326774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8326946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8327111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8327261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8327364Z skip: Need at least 2 CUDA devices (2.181s) 2022-11-23T01:34:53.8327370Z 2022-11-23T01:34:53.8327569Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8327643Z Ran 1 test in 2.181s 2022-11-23T01:34:53.8327649Z 2022-11-23T01:34:53.8327720Z OK (skipped=1) 2022-11-23T01:34:53.8327725Z 2022-11-23T01:34:53.8327808Z Generating XML reports... 2022-11-23T01:34:53.8328225Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013044.xml 2022-11-23T01:34:53.8328496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8328621Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8328889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8329025Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8329207Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps9dwi7s0 2022-11-23T01:34:53.8329403Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps9dwi7s0/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8329668Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8329676Z 2022-11-23T01:34:53.8329748Z Running tests... 2022-11-23T01:34:53.8329943Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8330210Z test_device_map_gpu_mixed_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31627 2022-11-23T01:34:53.8330365Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31628 2022-11-23T01:34:53.8330505Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31629 2022-11-23T01:34:53.8330656Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31630 2022-11-23T01:34:53.8330931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8331053Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8331333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8331471Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8331657Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi4xej_fd 2022-11-23T01:34:53.8331888Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi4xej_fd/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8332148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8332269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8332548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8332682Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8332868Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd5240w8h 2022-11-23T01:34:53.8333128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd5240w8h/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8333398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8333523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8333802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8333923Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8334109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpylg7o7zu 2022-11-23T01:34:53.8334305Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpylg7o7zu/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8334574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8334700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8334981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8335115Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8335304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppm5jn8p0 2022-11-23T01:34:53.8335485Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppm5jn8p0/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8335648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8335813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8335973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8336134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8336240Z skip: Need at least 2 CUDA devices (2.281s) 2022-11-23T01:34:53.8336246Z 2022-11-23T01:34:53.8336439Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8336516Z Ran 1 test in 2.281s 2022-11-23T01:34:53.8336521Z 2022-11-23T01:34:53.8336592Z OK (skipped=1) 2022-11-23T01:34:53.8336597Z 2022-11-23T01:34:53.8336667Z Generating XML reports... 2022-11-23T01:34:53.8337084Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013048.xml 2022-11-23T01:34:53.8337362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8337487Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8337766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8337902Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8338087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaqiehaua 2022-11-23T01:34:53.8338315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaqiehaua/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8338584Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8338590Z 2022-11-23T01:34:53.8338648Z Running tests... 2022-11-23T01:34:53.8338846Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8339116Z test_device_map_gpu_mixed_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31682 2022-11-23T01:34:53.8339272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31683 2022-11-23T01:34:53.8339424Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31684 2022-11-23T01:34:53.8339607Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31685 2022-11-23T01:34:53.8339880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8340005Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8340275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8340414Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8340598Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplgmz2tfe 2022-11-23T01:34:53.8340794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplgmz2tfe/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8341065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8341191Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8341470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8341607Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8341789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8s1t2nza 2022-11-23T01:34:53.8341970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8s1t2nza/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8342239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8342363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8342642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8342776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8342960Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_yhmo9xv 2022-11-23T01:34:53.8343152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_yhmo9xv/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8343425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8343545Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8343811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8343948Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8344132Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp23jwbs_n 2022-11-23T01:34:53.8344324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp23jwbs_n/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8344491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8344654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8344856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8345012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8345103Z skip: Need at least 2 CUDA devices (2.277s) 2022-11-23T01:34:53.8345107Z 2022-11-23T01:34:53.8345302Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8345379Z Ran 1 test in 2.277s 2022-11-23T01:34:53.8345383Z 2022-11-23T01:34:53.8345454Z OK (skipped=1) 2022-11-23T01:34:53.8345458Z 2022-11-23T01:34:53.8345541Z Generating XML reports... 2022-11-23T01:34:53.8345960Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013052.xml 2022-11-23T01:34:53.8346262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8346385Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8346668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8346789Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8346973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpponbdc2z 2022-11-23T01:34:53.8347169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpponbdc2z/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8347438Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8347443Z 2022-11-23T01:34:53.8347517Z Running tests... 2022-11-23T01:34:53.8347712Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8347979Z test_device_map_gpu_mixed_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31737 2022-11-23T01:34:53.8348138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31738 2022-11-23T01:34:53.8348278Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31739 2022-11-23T01:34:53.8348431Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31740 2022-11-23T01:34:53.8348702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8348825Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8349104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8349241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8349423Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_btv__u_ 2022-11-23T01:34:53.8349617Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_btv__u_/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8349887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8349996Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8350274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8350409Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8350590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8uyi0ggt 2022-11-23T01:34:53.8350785Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8uyi0ggt/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8351054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8351178Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8351486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8351621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8351792Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5s8tovqz 2022-11-23T01:34:53.8351985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5s8tovqz/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8352252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8352459Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8352779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8352914Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8353100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0dui8r93 2022-11-23T01:34:53.8353295Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0dui8r93/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8353445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8353607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8353769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8353929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8354036Z skip: Need at least 2 CUDA devices (2.179s) 2022-11-23T01:34:53.8354042Z 2022-11-23T01:34:53.8354237Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8354314Z Ran 1 test in 2.179s 2022-11-23T01:34:53.8354319Z 2022-11-23T01:34:53.8354391Z OK (skipped=1) 2022-11-23T01:34:53.8354395Z 2022-11-23T01:34:53.8354482Z Generating XML reports... 2022-11-23T01:34:53.8354886Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013056.xml 2022-11-23T01:34:53.8355160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8355283Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8355563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8355695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8355881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmo76e012 2022-11-23T01:34:53.8356076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmo76e012/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8356352Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8356357Z 2022-11-23T01:34:53.8356430Z Running tests... 2022-11-23T01:34:53.8356611Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8356880Z test_device_map_gpu_mixed_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31792 2022-11-23T01:34:53.8357035Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31793 2022-11-23T01:34:53.8357191Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31794 2022-11-23T01:34:53.8357347Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31795 2022-11-23T01:34:53.8357619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8357739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8358048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8358172Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8358357Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi631lqb0 2022-11-23T01:34:53.8358551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi631lqb0/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8358825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8358947Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8359251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8359385Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8359573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0yyhjs59 2022-11-23T01:34:53.8359765Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0yyhjs59/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8360020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8360140Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8360419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8360553Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8360907Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeuggbf0g 2022-11-23T01:34:53.8361106Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeuggbf0g/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8361383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8361506Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8361775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8361909Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8362092Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpud_iyb0m 2022-11-23T01:34:53.8362287Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpud_iyb0m/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8362453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8362618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8362780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8362944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8363049Z skip: Need at least 2 CUDA devices (2.279s) 2022-11-23T01:34:53.8363055Z 2022-11-23T01:34:53.8363238Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8363315Z Ran 1 test in 2.279s 2022-11-23T01:34:53.8363320Z 2022-11-23T01:34:53.8363393Z OK (skipped=1) 2022-11-23T01:34:53.8363397Z 2022-11-23T01:34:53.8363481Z Generating XML reports... 2022-11-23T01:34:53.8363902Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013100.xml 2022-11-23T01:34:53.8364176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8364302Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8364641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8364778Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8364950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe6p_i75k 2022-11-23T01:34:53.8365144Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe6p_i75k/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8365411Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8365416Z 2022-11-23T01:34:53.8365489Z Running tests... 2022-11-23T01:34:53.8365686Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8365999Z test_device_map_gpu_mixed_self_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31847 2022-11-23T01:34:53.8366159Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31848 2022-11-23T01:34:53.8366312Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31849 2022-11-23T01:34:53.8366450Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31850 2022-11-23T01:34:53.8366724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8366845Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8367123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8367258Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8367446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz11cixtq 2022-11-23T01:34:53.8367643Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz11cixtq/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8367917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8368038Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8368304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8368441Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8368628Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp007ggl3p 2022-11-23T01:34:53.8368822Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp007ggl3p/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8369090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8369211Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8369492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8369625Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8369795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3_3b0ufy 2022-11-23T01:34:53.8369989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3_3b0ufy/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8370257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8370379Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8370657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8370790Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8370974Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl70fjpb0 2022-11-23T01:34:53.8371204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl70fjpb0/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8371369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8371520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8371681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8371841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8371947Z skip: Need at least 2 CUDA devices (2.380s) 2022-11-23T01:34:53.8371979Z 2022-11-23T01:34:53.8372172Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8372248Z Ran 1 test in 2.380s 2022-11-23T01:34:53.8372253Z 2022-11-23T01:34:53.8372324Z OK (skipped=1) 2022-11-23T01:34:53.8372328Z 2022-11-23T01:34:53.8372413Z Generating XML reports... 2022-11-23T01:34:53.8372821Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013104.xml 2022-11-23T01:34:53.8373093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8373215Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8373494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8373629Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8373816Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzqwqc2dc 2022-11-23T01:34:53.8374012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzqwqc2dc/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8374284Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8374289Z 2022-11-23T01:34:53.8374361Z Running tests... 2022-11-23T01:34:53.8374543Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8374816Z test_device_map_gpu_mixed_self_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31902 2022-11-23T01:34:53.8374973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31903 2022-11-23T01:34:53.8375127Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31904 2022-11-23T01:34:53.8375281Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31905 2022-11-23T01:34:53.8375556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8375680Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8375962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8376098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8376271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnw27szdh 2022-11-23T01:34:53.8376469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnw27szdh/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8376738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8376858Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8377139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8377273Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8377487Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmf9d6u19 2022-11-23T01:34:53.8377684Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmf9d6u19/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8377951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8378060Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8378339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8378473Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8378657Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz349acsy 2022-11-23T01:34:53.8378879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz349acsy/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8379150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8379274Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8379552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8379672Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8379857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp__lq0ovi 2022-11-23T01:34:53.8380048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp__lq0ovi/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8380212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8380377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8380538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8380701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8380803Z skip: Need at least 2 CUDA devices (2.184s) 2022-11-23T01:34:53.8380808Z 2022-11-23T01:34:53.8381003Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8381065Z Ran 1 test in 2.184s 2022-11-23T01:34:53.8381070Z 2022-11-23T01:34:53.8381141Z OK (skipped=1) 2022-11-23T01:34:53.8381146Z 2022-11-23T01:34:53.8381229Z Generating XML reports... 2022-11-23T01:34:53.8381648Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013108.xml 2022-11-23T01:34:53.8381924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8382052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8382336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8382471Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8382641Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgt1d77dj 2022-11-23T01:34:53.8382840Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgt1d77dj/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8383105Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8383110Z 2022-11-23T01:34:53.8383182Z Running tests... 2022-11-23T01:34:53.8383383Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8383657Z test_device_map_gpu_mixed_self_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31957 2022-11-23T01:34:53.8383817Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31958 2022-11-23T01:34:53.8384000Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31959 2022-11-23T01:34:53.8384154Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31960 2022-11-23T01:34:53.8384418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8384542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8384824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8384962Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8385178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphx082jik 2022-11-23T01:34:53.8385373Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphx082jik/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8385647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8385771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8386053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8386174Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8386360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw5_0ov9y 2022-11-23T01:34:53.8386554Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw5_0ov9y/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8386827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8386950Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8387234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8387370Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8387552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1f3n3usd 2022-11-23T01:34:53.8387734Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1f3n3usd/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8388008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8388130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8388411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8388548Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8388734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsl6raurp 2022-11-23T01:34:53.8388929Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsl6raurp/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8389092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8389256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8389403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8389564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8389667Z skip: Need at least 2 CUDA devices (2.279s) 2022-11-23T01:34:53.8389672Z 2022-11-23T01:34:53.8389865Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8389941Z Ran 1 test in 2.279s 2022-11-23T01:34:53.8389946Z 2022-11-23T01:34:53.8390019Z OK (skipped=1) 2022-11-23T01:34:53.8390023Z 2022-11-23T01:34:53.8390107Z Generating XML reports... 2022-11-23T01:34:53.8390569Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013112.xml 2022-11-23T01:34:53.8390835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8390962Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8391245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8391381Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8391564Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq4etinif 2022-11-23T01:34:53.8391793Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq4etinif/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8392063Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8392068Z 2022-11-23T01:34:53.8392143Z Running tests... 2022-11-23T01:34:53.8392419Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8392686Z test_device_map_gpu_mixed_self_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32012 2022-11-23T01:34:53.8392846Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32013 2022-11-23T01:34:53.8393000Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32014 2022-11-23T01:34:53.8393153Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32015 2022-11-23T01:34:53.8393434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8393558Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8393845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8393980Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8394163Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8wj1xs8u 2022-11-23T01:34:53.8394345Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8wj1xs8u/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8394617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8394741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8395023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8395159Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8395346Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1buofo_o 2022-11-23T01:34:53.8395547Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1buofo_o/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8395819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8395928Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8396205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8396340Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8396526Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa9ua8vw0 2022-11-23T01:34:53.8396720Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa9ua8vw0/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8396990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8397149Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8397430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8397564Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8397735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuztmfen7 2022-11-23T01:34:53.8397927Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuztmfen7/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8398089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8398283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8398448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8398609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8398717Z skip: Need at least 2 CUDA devices (2.283s) 2022-11-23T01:34:53.8398723Z 2022-11-23T01:34:53.8398916Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8398978Z Ran 1 test in 2.283s 2022-11-23T01:34:53.8398994Z 2022-11-23T01:34:53.8399053Z OK (skipped=1) 2022-11-23T01:34:53.8399058Z 2022-11-23T01:34:53.8399141Z Generating XML reports... 2022-11-23T01:34:53.8399558Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013116.xml 2022-11-23T01:34:53.8399829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8399954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8400235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8400372Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8400553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ysr86yc 2022-11-23T01:34:53.8400855Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ysr86yc/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8401128Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8401133Z 2022-11-23T01:34:53.8401208Z Running tests... 2022-11-23T01:34:53.8401405Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8401679Z test_device_map_gpu_mixed_self_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32067 2022-11-23T01:34:53.8401839Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32068 2022-11-23T01:34:53.8401995Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32069 2022-11-23T01:34:53.8402149Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32070 2022-11-23T01:34:53.8402423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8402532Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8402814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8402951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8403140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgikq00bo 2022-11-23T01:34:53.8403341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgikq00bo/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8403612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8403782Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8404064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8404186Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8404368Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_us3hwao 2022-11-23T01:34:53.8404565Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_us3hwao/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8404837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8404995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8405271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8405406Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8405589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8avgj__4 2022-11-23T01:34:53.8405784Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8avgj__4/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8406041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8406167Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8406446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8406582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8406764Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8lskj5ft 2022-11-23T01:34:53.8406956Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8lskj5ft/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8407122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8407284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8407432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8407592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8407694Z skip: Need at least 2 CUDA devices (2.178s) 2022-11-23T01:34:53.8407701Z 2022-11-23T01:34:53.8407895Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8407971Z Ran 1 test in 2.178s 2022-11-23T01:34:53.8407976Z 2022-11-23T01:34:53.8408046Z OK (skipped=1) 2022-11-23T01:34:53.8408050Z 2022-11-23T01:34:53.8408132Z Generating XML reports... 2022-11-23T01:34:53.8408549Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013121.xml 2022-11-23T01:34:53.8408818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8408927Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8409208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8409341Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8409527Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplmp_i9f1 2022-11-23T01:34:53.8409724Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplmp_i9f1/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8409994Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8410000Z 2022-11-23T01:34:53.8410073Z Running tests... 2022-11-23T01:34:53.8410311Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8410584Z test_device_map_gpu_mixed_self_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32122 2022-11-23T01:34:53.8410727Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32123 2022-11-23T01:34:53.8410882Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32124 2022-11-23T01:34:53.8411035Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32125 2022-11-23T01:34:53.8411310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8411459Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8411739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8411873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8412058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpslz005y2 2022-11-23T01:34:53.8412241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpslz005y2/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8412512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8412635Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8412913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8413048Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8413231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcnwmezre 2022-11-23T01:34:53.8413428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcnwmezre/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8413697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8413821Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8414093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8414231Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8414414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjlilszf8 2022-11-23T01:34:53.8414607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjlilszf8/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8414877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8415000Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8415280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8415412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8415595Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvt23auy5 2022-11-23T01:34:53.8415776Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvt23auy5/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8415940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8416102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8416270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8416429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8416566Z skip: Need at least 2 CUDA devices (2.179s) 2022-11-23T01:34:53.8416572Z 2022-11-23T01:34:53.8416771Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8416845Z Ran 1 test in 2.179s 2022-11-23T01:34:53.8416850Z 2022-11-23T01:34:53.8416909Z OK (skipped=1) 2022-11-23T01:34:53.8416927Z 2022-11-23T01:34:53.8416998Z Generating XML reports... 2022-11-23T01:34:53.8417417Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013125.xml 2022-11-23T01:34:53.8417699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8417856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8418138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8418270Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8418456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxqamdxdh 2022-11-23T01:34:53.8418651Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxqamdxdh/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8418904Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8418909Z 2022-11-23T01:34:53.8418979Z Running tests... 2022-11-23T01:34:53.8419173Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8419445Z test_device_map_gpu_mixed_self_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32177 2022-11-23T01:34:53.8419603Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32178 2022-11-23T01:34:53.8419757Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32179 2022-11-23T01:34:53.8419914Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32180 2022-11-23T01:34:53.8420187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8420308Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8420577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8420714Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8420899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqk1yolda 2022-11-23T01:34:53.8421098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqk1yolda/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8421370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8421496Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8421775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8421907Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8422078Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprbo2eoy7 2022-11-23T01:34:53.8422275Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprbo2eoy7/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8422545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8422672Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8422948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8423081Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8423295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpucpsshb0 2022-11-23T01:34:53.8423500Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpucpsshb0/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8423769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8423879Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8424157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8424290Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8424506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzcv7ullt 2022-11-23T01:34:53.8424699Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzcv7ullt/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8424864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8425026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8425186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8425334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8425435Z skip: Need at least 2 CUDA devices (2.278s) 2022-11-23T01:34:53.8425441Z 2022-11-23T01:34:53.8425637Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8425713Z Ran 1 test in 2.279s 2022-11-23T01:34:53.8425720Z 2022-11-23T01:34:53.8425792Z OK (skipped=1) 2022-11-23T01:34:53.8425797Z 2022-11-23T01:34:53.8425882Z Generating XML reports... 2022-11-23T01:34:53.8426297Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013129.xml 2022-11-23T01:34:53.8426573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8426698Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8426967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8427104Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8427287Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf0_i5yvq 2022-11-23T01:34:53.8427480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf0_i5yvq/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8427751Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8427756Z 2022-11-23T01:34:53.8427826Z Running tests... 2022-11-23T01:34:53.8428023Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8428292Z test_device_map_gpu_mixed_self_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32232 2022-11-23T01:34:53.8428446Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32233 2022-11-23T01:34:53.8428587Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32234 2022-11-23T01:34:53.8428741Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32235 2022-11-23T01:34:53.8429016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8429144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8429425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8429561Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8429776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl7v0m800 2022-11-23T01:34:53.8429970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl7v0m800/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8430231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8430352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8430630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8430792Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8430975Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkgbz83ce 2022-11-23T01:34:53.8431169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkgbz83ce/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8431440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8431560Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8431842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8431966Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8432149Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ymnzb4w 2022-11-23T01:34:53.8432416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ymnzb4w/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8432694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8432815Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8433097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8433232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8433418Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe0ws3xtz 2022-11-23T01:34:53.8433599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe0ws3xtz/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8433769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8433935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8434103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8434268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8434378Z skip: Need at least 2 CUDA devices (2.286s) 2022-11-23T01:34:53.8434382Z 2022-11-23T01:34:53.8434583Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8434662Z Ran 1 test in 2.286s 2022-11-23T01:34:53.8434667Z 2022-11-23T01:34:53.8434738Z OK (skipped=1) 2022-11-23T01:34:53.8434743Z 2022-11-23T01:34:53.8434814Z Generating XML reports... 2022-11-23T01:34:53.8435234Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013133.xml 2022-11-23T01:34:53.8435505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8435628Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8435912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8436048Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8436336Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp72tluh7n 2022-11-23T01:34:53.8436536Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp72tluh7n/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8436806Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8436811Z 2022-11-23T01:34:53.8436871Z Running tests... 2022-11-23T01:34:53.8437067Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8437340Z test_device_map_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32287 2022-11-23T01:34:53.8437536Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32288 2022-11-23T01:34:53.8437693Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32289 2022-11-23T01:34:53.8437848Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32290 2022-11-23T01:34:53.8438128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8438250Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8438517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8438651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8438833Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9mvz8a8a 2022-11-23T01:34:53.8439026Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9mvz8a8a/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8439303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8439424Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8439705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8439838Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8440022Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpknvw6g4c 2022-11-23T01:34:53.8440205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpknvw6g4c/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8440472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8440699Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8440986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8441125Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8441314Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjakjxp98 2022-11-23T01:34:53.8441514Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjakjxp98/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8441787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8441911Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8442180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8442315Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8442502Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp55rnlc3j 2022-11-23T01:34:53.8442695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp55rnlc3j/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8442859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8443071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8443236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8443399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8443490Z skip: Need at least 2 CUDA devices (2.281s) 2022-11-23T01:34:53.8443496Z 2022-11-23T01:34:53.8443694Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8443769Z Ran 1 test in 2.281s 2022-11-23T01:34:53.8443775Z 2022-11-23T01:34:53.8443847Z OK (skipped=1) 2022-11-23T01:34:53.8443888Z 2022-11-23T01:34:53.8443973Z Generating XML reports... 2022-11-23T01:34:53.8444397Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013137.xml 2022-11-23T01:34:53.8444673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8444796Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8445078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8445203Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8445386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_6je_h69 2022-11-23T01:34:53.8445578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_6je_h69/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8445847Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8445852Z 2022-11-23T01:34:53.8445925Z Running tests... 2022-11-23T01:34:53.8446123Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8446412Z test_device_map_gpu_non_default_to_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32342 2022-11-23T01:34:53.8446568Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32343 2022-11-23T01:34:53.8446723Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32344 2022-11-23T01:34:53.8446864Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32345 2022-11-23T01:34:53.8447137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8447261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8447545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8447679Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8447868Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfkck_ljr 2022-11-23T01:34:53.8448066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfkck_ljr/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8448338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8448448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8448730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8448863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8449048Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbszm_c__ 2022-11-23T01:34:53.8449242Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbszm_c__/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8449542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8449666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8449942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8450075Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8450247Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpum0gajqg 2022-11-23T01:34:53.8450443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpum0gajqg/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8450746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8450868Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8451150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8451284Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8451471Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpayx__rv2 2022-11-23T01:34:53.8451664Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpayx__rv2/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8451815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8451977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8452139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8452300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8452405Z skip: Need at least 2 CUDA devices (2.279s) 2022-11-23T01:34:53.8452414Z 2022-11-23T01:34:53.8452616Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8452691Z Ran 1 test in 2.279s 2022-11-23T01:34:53.8452696Z 2022-11-23T01:34:53.8452768Z OK (skipped=1) 2022-11-23T01:34:53.8452772Z 2022-11-23T01:34:53.8452854Z Generating XML reports... 2022-11-23T01:34:53.8453262Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013141.xml 2022-11-23T01:34:53.8453536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8453660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8453940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8454075Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8454259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4k1lquuk 2022-11-23T01:34:53.8454458Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4k1lquuk/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8454727Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8454732Z 2022-11-23T01:34:53.8454803Z Running tests... 2022-11-23T01:34:53.8454984Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8455266Z test_device_map_gpu_to_cpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32397 2022-11-23T01:34:53.8455427Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32398 2022-11-23T01:34:53.8455579Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32399 2022-11-23T01:34:53.8455732Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32400 2022-11-23T01:34:53.8456033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8456159Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8456442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8456564Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8456751Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp16dmg_n9 2022-11-23T01:34:53.8456947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp16dmg_n9/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8457270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8457396Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8457675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8457809Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8457992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplkip13u9 2022-11-23T01:34:53.8458185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplkip13u9/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8458442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8458567Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8458848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8458986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8459167Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_xt8l33m 2022-11-23T01:34:53.8459358Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_xt8l33m/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8459629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8459753Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8460016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8460149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8460334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbru_d1cx 2022-11-23T01:34:53.8460527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbru_d1cx/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8460691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8460856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8461019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8461179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8461278Z skip: Need at least 1 CUDA device (2.278s) 2022-11-23T01:34:53.8461284Z 2022-11-23T01:34:53.8461465Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8461540Z Ran 1 test in 2.278s 2022-11-23T01:34:53.8461545Z 2022-11-23T01:34:53.8461616Z OK (skipped=1) 2022-11-23T01:34:53.8461620Z 2022-11-23T01:34:53.8461704Z Generating XML reports... 2022-11-23T01:34:53.8462121Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013145.xml 2022-11-23T01:34:53.8462424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8462549Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8462826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8462959Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8463130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplznxjjuw 2022-11-23T01:34:53.8463328Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplznxjjuw/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8463595Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8463627Z 2022-11-23T01:34:53.8463701Z Running tests... 2022-11-23T01:34:53.8463896Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8464178Z test_device_map_gpu_to_cpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32452 2022-11-23T01:34:53.8464335Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32453 2022-11-23T01:34:53.8464486Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32454 2022-11-23T01:34:53.8464623Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32455 2022-11-23T01:34:53.8464897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8465021Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8465305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8465437Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8465627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiecth2yt 2022-11-23T01:34:53.8465824Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiecth2yt/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8466096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8466216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8466479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8466615Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8466796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq79p3rmz 2022-11-23T01:34:53.8466994Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq79p3rmz/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8467261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8467383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8467658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8467793Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8467977Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfci3twyp 2022-11-23T01:34:53.8468161Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfci3twyp/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8468429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8468554Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8468833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8468999Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8469185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppn7chfpb 2022-11-23T01:34:53.8469378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppn7chfpb/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8469540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8469689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8469853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8470042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8470145Z skip: Need at least 2 CUDA devices (2.180s) 2022-11-23T01:34:53.8470150Z 2022-11-23T01:34:53.8470345Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8470423Z Ran 1 test in 2.180s 2022-11-23T01:34:53.8470428Z 2022-11-23T01:34:53.8470500Z OK (skipped=1) 2022-11-23T01:34:53.8470505Z 2022-11-23T01:34:53.8470586Z Generating XML reports... 2022-11-23T01:34:53.8471006Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013149.xml 2022-11-23T01:34:53.8471267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8471390Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8471666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8471803Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8471987Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj4t27768 2022-11-23T01:34:53.8472182Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj4t27768/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8472533Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8472541Z 2022-11-23T01:34:53.8472614Z Running tests... 2022-11-23T01:34:53.8472798Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8473061Z test_device_maps_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32507 2022-11-23T01:34:53.8473222Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32508 2022-11-23T01:34:53.8473381Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32509 2022-11-23T01:34:53.8473532Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32510 2022-11-23T01:34:53.8473811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8473933Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8474219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8474355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8474526Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxuhhld53 2022-11-23T01:34:53.8474725Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxuhhld53/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8474997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8475124Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8475405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8475574Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8475760Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp75r45fgr 2022-11-23T01:34:53.8475953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp75r45fgr/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8476214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8476340Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8476615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8476777Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8476959Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv9c8l8c0 2022-11-23T01:34:53.8477151Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv9c8l8c0/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8477419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8477542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8477812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8477946Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8478129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplg6516zg 2022-11-23T01:34:53.8478324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplg6516zg/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8478487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8478651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8478815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8478972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8479062Z skip: Need at least 2 CUDA devices (2.279s) 2022-11-23T01:34:53.8479068Z 2022-11-23T01:34:53.8479263Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8479337Z Ran 1 test in 2.280s 2022-11-23T01:34:53.8479342Z 2022-11-23T01:34:53.8479413Z OK (skipped=1) 2022-11-23T01:34:53.8479417Z 2022-11-23T01:34:53.8479504Z Generating XML reports... 2022-11-23T01:34:53.8479924Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013153.xml 2022-11-23T01:34:53.8480198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8480322Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8480711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8480836Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8481019Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp06ll0_3c 2022-11-23T01:34:53.8481211Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp06ll0_3c/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8481480Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8481487Z 2022-11-23T01:34:53.8481558Z Running tests... 2022-11-23T01:34:53.8481751Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8482078Z test_device_maps_in_options (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32562 2022-11-23T01:34:53.8482238Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32563 2022-11-23T01:34:53.8482378Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32564 2022-11-23T01:34:53.8482530Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32565 2022-11-23T01:34:53.8482806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8482929Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8483211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8483385Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8483569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz727e2y3 2022-11-23T01:34:53.8483768Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz727e2y3/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8484042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8484151Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8484430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8484563Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8484749Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcj2zp9_q 2022-11-23T01:34:53.8484951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcj2zp9_q/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8485220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8485343Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8485621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8485754Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8485926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfa78so57 2022-11-23T01:34:53.8486119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfa78so57/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8486390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8486514Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8486791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8486924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8487109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo99blev2 2022-11-23T01:34:53.8487300Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo99blev2/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8487451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8487613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8487772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8487933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8488041Z skip: Need at least 2 CUDA devices (2.279s) 2022-11-23T01:34:53.8488046Z 2022-11-23T01:34:53.8488241Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8488316Z Ran 1 test in 2.279s 2022-11-23T01:34:53.8488321Z 2022-11-23T01:34:53.8488422Z OK (skipped=1) 2022-11-23T01:34:53.8488427Z 2022-11-23T01:34:53.8488511Z Generating XML reports... 2022-11-23T01:34:53.8488917Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013157.xml 2022-11-23T01:34:53.8489191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8489313Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8489591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8489753Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8489936Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptofhe5xv 2022-11-23T01:34:53.8490130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptofhe5xv/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8490399Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8490404Z 2022-11-23T01:34:53.8490477Z Running tests... 2022-11-23T01:34:53.8490659Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8490943Z test_device_maps_invalid_max_local_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32617 2022-11-23T01:34:53.8491097Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32618 2022-11-23T01:34:53.8491249Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32619 2022-11-23T01:34:53.8491399Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32620 2022-11-23T01:34:53.8491672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8491796Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8492072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8492193Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8492378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzkldbtz8 2022-11-23T01:34:53.8492574Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzkldbtz8/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8492845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8492967Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8493246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8493380Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8493562Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1q7j3q09 2022-11-23T01:34:53.8493752Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1q7j3q09/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8494011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8494131Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8494409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8494547Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8494730Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdcy7pad2 2022-11-23T01:34:53.8494923Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdcy7pad2/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8495225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8495348Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8495619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8495752Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8495935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcqchu_7d 2022-11-23T01:34:53.8496129Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcqchu_7d/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8496320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8496484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8496647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8496808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8496907Z skip: Need at least 1 CUDA device (2.184s) 2022-11-23T01:34:53.8496912Z 2022-11-23T01:34:53.8497094Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8497170Z Ran 1 test in 2.184s 2022-11-23T01:34:53.8497175Z 2022-11-23T01:34:53.8497245Z OK (skipped=1) 2022-11-23T01:34:53.8497250Z 2022-11-23T01:34:53.8497331Z Generating XML reports... 2022-11-23T01:34:53.8497748Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013201.xml 2022-11-23T01:34:53.8498023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8498144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8498426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8498558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8498730Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptwsvwpvd 2022-11-23T01:34:53.8498926Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptwsvwpvd/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8499196Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8499201Z 2022-11-23T01:34:53.8499279Z Running tests... 2022-11-23T01:34:53.8499472Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8499755Z test_device_maps_invalid_max_remote_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32672 2022-11-23T01:34:53.8499913Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32673 2022-11-23T01:34:53.8500066Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32674 2022-11-23T01:34:53.8500205Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32675 2022-11-23T01:34:53.8500477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8500598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8500876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8501011Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8501195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn3dnttuf 2022-11-23T01:34:53.8501425Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn3dnttuf/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8501700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8501822Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8502087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8502221Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8502403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9nlfd60x 2022-11-23T01:34:53.8502596Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9nlfd60x/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8502902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8503024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8503302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8503435Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8503605Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9h70wob7 2022-11-23T01:34:53.8503796Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9h70wob7/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8504064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8504184Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8504464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8504597Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8504781Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfk1v4jg8 2022-11-23T01:34:53.8504976Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfk1v4jg8/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8505136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8505286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8505446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8505604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8505706Z skip: Need at least 1 CUDA device (2.279s) 2022-11-23T01:34:53.8505711Z 2022-11-23T01:34:53.8505905Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8505978Z Ran 1 test in 2.279s 2022-11-23T01:34:53.8505983Z 2022-11-23T01:34:53.8506053Z OK (skipped=1) 2022-11-23T01:34:53.8506060Z 2022-11-23T01:34:53.8506141Z Generating XML reports... 2022-11-23T01:34:53.8506557Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013205.xml 2022-11-23T01:34:53.8506817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8506938Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8507220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8507353Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8507541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa1i3rb46 2022-11-23T01:34:53.8507733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa1i3rb46/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8508032Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8508038Z 2022-11-23T01:34:53.8508111Z Running tests... 2022-11-23T01:34:53.8508294Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8508569Z test_device_maps_invalid_min_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32727 2022-11-23T01:34:53.8508723Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32728 2022-11-23T01:34:53.8508878Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32729 2022-11-23T01:34:53.8509062Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32730 2022-11-23T01:34:53.8509336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8509460Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8509738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8509873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8510044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmzlnai91 2022-11-23T01:34:53.8510237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmzlnai91/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8510506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8510631Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8510909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8511042Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8511226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp42sjllpi 2022-11-23T01:34:53.8511423Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp42sjllpi/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8511696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8511804Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8512085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8512218Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8512478Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfxfbausb 2022-11-23T01:34:53.8512675Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfxfbausb/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8512945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8513069Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8513347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8513467Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8513650Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd2kio2nb 2022-11-23T01:34:53.8513840Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd2kio2nb/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8514005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8514166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8514366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8514529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8514632Z skip: Need at least 1 CUDA device (2.178s) 2022-11-23T01:34:53.8514636Z 2022-11-23T01:34:53.8514829Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8514891Z Ran 1 test in 2.178s 2022-11-23T01:34:53.8514897Z 2022-11-23T01:34:53.8514969Z OK (skipped=1) 2022-11-23T01:34:53.8514973Z 2022-11-23T01:34:53.8515056Z Generating XML reports... 2022-11-23T01:34:53.8515471Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013210.xml 2022-11-23T01:34:53.8515772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8515894Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8516177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8516314Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8516489Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdb1c30mt 2022-11-23T01:34:53.8516684Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdb1c30mt/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8516950Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8516955Z 2022-11-23T01:34:53.8517027Z Running tests... 2022-11-23T01:34:53.8517225Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8517492Z test_device_maps_many_to_one (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 314 2022-11-23T01:34:53.8517646Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 315 2022-11-23T01:34:53.8517797Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 316 2022-11-23T01:34:53.8517945Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 317 2022-11-23T01:34:53.8518203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8518324Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8518602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8518737Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8518923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuwsq2278 2022-11-23T01:34:53.8519116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuwsq2278/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8519393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8519514Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8519789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8519911Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8520093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn9nl4yk3 2022-11-23T01:34:53.8520288Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn9nl4yk3/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8520560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8520793Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8521127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8521262Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8521445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8tu2n_1z 2022-11-23T01:34:53.8521624Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8tu2n_1z/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8521894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8522017Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8522334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8522467Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8522655Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjy73x_c5 2022-11-23T01:34:53.8522848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjy73x_c5/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8523010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8523173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8523320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8523481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8523588Z skip: Need at least 2 CUDA devices (2.180s) 2022-11-23T01:34:53.8523596Z 2022-11-23T01:34:53.8523791Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8523865Z Ran 1 test in 2.180s 2022-11-23T01:34:53.8523870Z 2022-11-23T01:34:53.8523941Z OK (skipped=1) 2022-11-23T01:34:53.8523946Z 2022-11-23T01:34:53.8524029Z Generating XML reports... 2022-11-23T01:34:53.8524448Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013214.xml 2022-11-23T01:34:53.8524723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8524834Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8525112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8525246Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8525436Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt4xp__px 2022-11-23T01:34:53.8525629Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt4xp__px/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8525895Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8525900Z 2022-11-23T01:34:53.8525971Z Running tests... 2022-11-23T01:34:53.8526168Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8526425Z test_device_maps_missing_config (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 369 2022-11-23T01:34:53.8526577Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 370 2022-11-23T01:34:53.8526725Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 371 2022-11-23T01:34:53.8526874Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 372 2022-11-23T01:34:53.8527150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8527272Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8527598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8527737Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8527918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpczmrmk15 2022-11-23T01:34:53.8528101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpczmrmk15/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8528375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8528496Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8528806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8528941Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8529128Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplv58j_0_ 2022-11-23T01:34:53.8529318Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplv58j_0_/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8529587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8529696Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8529975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8530107Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8530291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvqgbrpvq 2022-11-23T01:34:53.8530488Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvqgbrpvq/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8530760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8530882Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8531162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8531295Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8531466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg80a5n9d 2022-11-23T01:34:53.8531657Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg80a5n9d/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8531819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8531983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8532142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8532307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8532407Z skip: Need at least 1 CUDA device (2.180s) 2022-11-23T01:34:53.8532412Z 2022-11-23T01:34:53.8532605Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8532667Z Ran 1 test in 2.180s 2022-11-23T01:34:53.8532683Z 2022-11-23T01:34:53.8532741Z OK (skipped=1) 2022-11-23T01:34:53.8532745Z 2022-11-23T01:34:53.8532828Z Generating XML reports... 2022-11-23T01:34:53.8533246Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013218.xml 2022-11-23T01:34:53.8533523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8533645Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8533956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8534091Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8534276Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_hk0bty3 2022-11-23T01:34:53.8534456Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_hk0bty3/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8534722Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8534727Z 2022-11-23T01:34:53.8534798Z Running tests... 2022-11-23T01:34:53.8534990Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8535330Z test_device_maps_missing_config_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 424 2022-11-23T01:34:53.8535483Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 425 2022-11-23T01:34:53.8535635Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 426 2022-11-23T01:34:53.8535785Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 427 2022-11-23T01:34:53.8536055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8536166Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8536448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8536583Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8536769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf47_g0td 2022-11-23T01:34:53.8536966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf47_g0td/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8537238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8537360Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8537638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8537759Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8537941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0gloeb1k 2022-11-23T01:34:53.8538135Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0gloeb1k/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8538404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8538527Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8538805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8538940Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8539123Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf0ods9u0 2022-11-23T01:34:53.8539316Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf0ods9u0/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8539574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8539696Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8539974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8540108Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8540292Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz82tvgqb 2022-11-23T01:34:53.8540517Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz82tvgqb/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8540682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8540843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8541004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8541152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8541252Z skip: Need at least 1 CUDA device (2.181s) 2022-11-23T01:34:53.8541258Z 2022-11-23T01:34:53.8541451Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8541555Z Ran 1 test in 2.181s 2022-11-23T01:34:53.8541561Z 2022-11-23T01:34:53.8541632Z OK (skipped=1) 2022-11-23T01:34:53.8541637Z 2022-11-23T01:34:53.8541721Z Generating XML reports... 2022-11-23T01:34:53.8542143Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013222.xml 2022-11-23T01:34:53.8542417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8542529Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8542807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8542940Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8543121Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_328my1j 2022-11-23T01:34:53.8543313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_328my1j/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8543585Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8543589Z 2022-11-23T01:34:53.8543664Z Running tests... 2022-11-23T01:34:53.8543859Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8544139Z test_device_maps_missing_config_not_timeout (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 479 2022-11-23T01:34:53.8544280Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 480 2022-11-23T01:34:53.8544431Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 481 2022-11-23T01:34:53.8544578Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 482 2022-11-23T01:34:53.8544854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8544977Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8545258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8545392Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8545572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5jwuwgy9 2022-11-23T01:34:53.8545841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8545950Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8546229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8546367Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8546552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7zzcodz5 2022-11-23T01:34:53.8546747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5jwuwgy9/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8546966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7zzcodz5/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8547238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8547360Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8547626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8547759Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8547941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxgv2xyy4 2022-11-23T01:34:53.8548166Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxgv2xyy4/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8548436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8548559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8548840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8548975Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8549156Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp717b_307 2022-11-23T01:34:53.8549332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp717b_307/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8549497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8549663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8549824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8549984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8550085Z skip: Need at least 1 CUDA device (2.182s) 2022-11-23T01:34:53.8550091Z 2022-11-23T01:34:53.8550284Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8550357Z Ran 1 test in 2.182s 2022-11-23T01:34:53.8550362Z 2022-11-23T01:34:53.8550419Z OK (skipped=1) 2022-11-23T01:34:53.8550436Z 2022-11-23T01:34:53.8550506Z Generating XML reports... 2022-11-23T01:34:53.8550926Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013226.xml 2022-11-23T01:34:53.8551199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8551325Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8551603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8551738Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8551922Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb139c5gf 2022-11-23T01:34:53.8552113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb139c5gf/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8552444Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8552463Z 2022-11-23T01:34:53.8552523Z Running tests... 2022-11-23T01:34:53.8552718Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8552994Z test_device_maps_missing_config_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 534 2022-11-23T01:34:53.8553149Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 535 2022-11-23T01:34:53.8553298Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 536 2022-11-23T01:34:53.8553486Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 537 2022-11-23T01:34:53.8553764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8553886Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8554152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8554286Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8554470Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp22azy86p 2022-11-23T01:34:53.8554696Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp22azy86p/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8554964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8555087Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8555362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8555496Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8555670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpndag5e94 2022-11-23T01:34:53.8555863Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpndag5e94/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8556133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8556256Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8556536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8556674Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8556857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuk21zfld 2022-11-23T01:34:53.8557051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuk21zfld/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8557322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8557430Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8557708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8557844Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8558028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgxivgtgk 2022-11-23T01:34:53.8558221Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgxivgtgk/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8558387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8558549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8558711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8558858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8558958Z skip: Need at least 1 CUDA device (2.284s) 2022-11-23T01:34:53.8558963Z 2022-11-23T01:34:53.8559158Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8559235Z Ran 1 test in 2.285s 2022-11-23T01:34:53.8559240Z 2022-11-23T01:34:53.8559310Z OK (skipped=1) 2022-11-23T01:34:53.8559314Z 2022-11-23T01:34:53.8559397Z Generating XML reports... 2022-11-23T01:34:53.8559843Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013230.xml 2022-11-23T01:34:53.8560119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8560240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8560507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8560753Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8560938Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprh5lqm5d 2022-11-23T01:34:53.8561179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprh5lqm5d/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8561449Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8561454Z 2022-11-23T01:34:53.8561525Z Running tests... 2022-11-23T01:34:53.8561721Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8562011Z test_device_maps_missing_config_remote_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 589 2022-11-23T01:34:53.8562163Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 590 2022-11-23T01:34:53.8562300Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 591 2022-11-23T01:34:53.8562449Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 592 2022-11-23T01:34:53.8562724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8562852Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8563134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8563269Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8563452Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy06fgec4 2022-11-23T01:34:53.8563649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy06fgec4/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8563918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8564028Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8564304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8564443Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8564628Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpow32h7ez 2022-11-23T01:34:53.8564824Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpow32h7ez/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8565093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8565214Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8565492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8565613Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8565794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpukqnun6q 2022-11-23T01:34:53.8565986Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpukqnun6q/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8566256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8566378Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8566699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8566834Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8567019Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppqx0jlf2 2022-11-23T01:34:53.8567211Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppqx0jlf2/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8567361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8567523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8567713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8567874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8567975Z skip: Need at least 1 CUDA device (2.184s) 2022-11-23T01:34:53.8567981Z 2022-11-23T01:34:53.8568177Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8568251Z Ran 1 test in 2.184s 2022-11-23T01:34:53.8568256Z 2022-11-23T01:34:53.8568326Z OK (skipped=1) 2022-11-23T01:34:53.8568330Z 2022-11-23T01:34:53.8568401Z Generating XML reports... 2022-11-23T01:34:53.8568818Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013234.xml 2022-11-23T01:34:53.8569089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8569212Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8569491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8569623Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8569808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5jsbikkq 2022-11-23T01:34:53.8570002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5jsbikkq/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8570269Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8570274Z 2022-11-23T01:34:53.8570332Z Running tests... 2022-11-23T01:34:53.8570523Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8570801Z test_device_maps_missing_config_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 645 2022-11-23T01:34:53.8570954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 646 2022-11-23T01:34:53.8571102Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 647 2022-11-23T01:34:53.8571253Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 648 2022-11-23T01:34:53.8571524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8571644Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8571922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8572044Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8572226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp10p_7h9z 2022-11-23T01:34:53.8572422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp10p_7h9z/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8572693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8572857Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8573135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8573267Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8573450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphfjw1p3u 2022-11-23T01:34:53.8573630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphfjw1p3u/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8573901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8574050Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8574332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8574467Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8574650Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4xb_af4f 2022-11-23T01:34:53.8574845Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4xb_af4f/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8575113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8575234Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8575502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8575634Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8575820Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvnbvq1h_ 2022-11-23T01:34:53.8576016Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvnbvq1h_/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8576180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8576342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8576501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8576661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8576748Z skip: Need at least 1 CUDA device (2.279s) 2022-11-23T01:34:53.8576765Z 2022-11-23T01:34:53.8576945Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8577022Z Ran 1 test in 2.279s 2022-11-23T01:34:53.8577029Z 2022-11-23T01:34:53.8577102Z OK (skipped=1) 2022-11-23T01:34:53.8577107Z 2022-11-23T01:34:53.8577189Z Generating XML reports... 2022-11-23T01:34:53.8577606Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013238.xml 2022-11-23T01:34:53.8577880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8578001Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8578283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8578404Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8578585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6tsx7u49 2022-11-23T01:34:53.8578776Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6tsx7u49/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8579049Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8579054Z 2022-11-23T01:34:53.8579125Z Running tests... 2022-11-23T01:34:53.8579349Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8579636Z test_device_maps_missing_config_response_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 701 2022-11-23T01:34:53.8579788Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 702 2022-11-23T01:34:53.8579939Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 703 2022-11-23T01:34:53.8580076Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 704 2022-11-23T01:34:53.8580350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8580502Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8580785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8580918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8581104Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxojt5vh6 2022-11-23T01:34:53.8581300Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxojt5vh6/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8581573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8581681Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8581960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8582096Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8582279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpif8f9n1m 2022-11-23T01:34:53.8582472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpif8f9n1m/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8582744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8582864Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8583141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8583273Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8583443Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6d9rapvk 2022-11-23T01:34:53.8583636Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6d9rapvk/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8583907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8584024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8584304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8584437Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8584619Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprpr1wtfg 2022-11-23T01:34:53.8584813Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprpr1wtfg/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8584975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8585124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8585287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8585450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8585548Z skip: Need at least 1 CUDA device (2.278s) 2022-11-23T01:34:53.8585553Z 2022-11-23T01:34:53.8585775Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8585852Z Ran 1 test in 2.279s 2022-11-23T01:34:53.8585857Z 2022-11-23T01:34:53.8585927Z OK (skipped=1) 2022-11-23T01:34:53.8585932Z 2022-11-23T01:34:53.8586013Z Generating XML reports... 2022-11-23T01:34:53.8586422Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013242.xml 2022-11-23T01:34:53.8586693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8586815Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8587122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8587255Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8587440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw5yeyzh3 2022-11-23T01:34:53.8587632Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw5yeyzh3/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8587899Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8587904Z 2022-11-23T01:34:53.8587974Z Running tests... 2022-11-23T01:34:53.8588156Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8588419Z test_device_maps_multi_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 756 2022-11-23T01:34:53.8588574Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 757 2022-11-23T01:34:53.8588725Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 758 2022-11-23T01:34:53.8588873Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 759 2022-11-23T01:34:53.8589146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8589269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8589551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8589671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8589854Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcxaogwhs 2022-11-23T01:34:53.8590048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcxaogwhs/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8590321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8590442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8590722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8590856Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8591042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw4mup6s_ 2022-11-23T01:34:53.8591238Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw4mup6s_/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8591494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8591619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8591900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8592033Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8592244Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk7s5wg62 2022-11-23T01:34:53.8592508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk7s5wg62/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8592785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8592908Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8593186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8593306Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8593488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3u7g7_yt 2022-11-23T01:34:53.8593712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3u7g7_yt/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8593877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8594043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8594204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8594363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8594465Z skip: Need at least 2 CUDA devices (2.279s) 2022-11-23T01:34:53.8594470Z 2022-11-23T01:34:53.8594651Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8594725Z Ran 1 test in 2.279s 2022-11-23T01:34:53.8594730Z 2022-11-23T01:34:53.8594800Z OK (skipped=1) 2022-11-23T01:34:53.8594806Z 2022-11-23T01:34:53.8594889Z Generating XML reports... 2022-11-23T01:34:53.8595307Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013246.xml 2022-11-23T01:34:53.8595581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8595704Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8595982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8596115Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8596287Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpybdy_4r5 2022-11-23T01:34:53.8596480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpybdy_4r5/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8596748Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8596753Z 2022-11-23T01:34:53.8596825Z Running tests... 2022-11-23T01:34:53.8597019Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8597290Z test_device_maps_multi_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 811 2022-11-23T01:34:53.8597443Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 812 2022-11-23T01:34:53.8597592Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 813 2022-11-23T01:34:53.8597740Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 814 2022-11-23T01:34:53.8598001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8598124Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8598405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8598540Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8598752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd2y404ek 2022-11-23T01:34:53.8598947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd2y404ek/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8599217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8599340Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8599603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8599737Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8599951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg499v995 2022-11-23T01:34:53.8600146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg499v995/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8600416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8600538Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8600924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8601059Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8601241Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9av1f4rt 2022-11-23T01:34:53.8601422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9av1f4rt/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8601689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8601813Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8602089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8602224Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8602408Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplk4_o3x6 2022-11-23T01:34:53.8602598Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplk4_o3x6/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8602760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8602910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8603071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8603233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8603336Z skip: Need at least 2 CUDA devices (2.179s) 2022-11-23T01:34:53.8603341Z 2022-11-23T01:34:53.8603537Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8603614Z Ran 1 test in 2.179s 2022-11-23T01:34:53.8603618Z 2022-11-23T01:34:53.8603689Z OK (skipped=1) 2022-11-23T01:34:53.8603694Z 2022-11-23T01:34:53.8603776Z Generating XML reports... 2022-11-23T01:34:53.8604196Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013250.xml 2022-11-23T01:34:53.8604457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8604579Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8604857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8604992Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8605174Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmput0uxt0h 2022-11-23T01:34:53.8605419Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmput0uxt0h/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8605687Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8605692Z 2022-11-23T01:34:53.8605764Z Running tests... 2022-11-23T01:34:53.8605958Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8606211Z test_device_maps_one_to_many (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 866 2022-11-23T01:34:53.8606365Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 867 2022-11-23T01:34:53.8606552Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 868 2022-11-23T01:34:53.8606700Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 869 2022-11-23T01:34:53.8606973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8607095Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8607374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8607508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8607678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplct57lre 2022-11-23T01:34:53.8607870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplct57lre/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8608141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8608262Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8608544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8608678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8608861Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbg6hu03i 2022-11-23T01:34:53.8609058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbg6hu03i/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8609328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8609437Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8609715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8609850Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8610033Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppzkotc0m 2022-11-23T01:34:53.8610228Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppzkotc0m/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8610495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8610617Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8610894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8611013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8611194Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6owy_yzu 2022-11-23T01:34:53.8611389Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6owy_yzu/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8611557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8611749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8611911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8612071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8612173Z skip: Need at least 2 CUDA devices (2.279s) 2022-11-23T01:34:53.8612178Z 2022-11-23T01:34:53.8612373Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8612435Z Ran 1 test in 2.279s 2022-11-23T01:34:53.8612440Z 2022-11-23T01:34:53.8612509Z OK (skipped=1) 2022-11-23T01:34:53.8612514Z 2022-11-23T01:34:53.8612596Z Generating XML reports... 2022-11-23T01:34:53.8613046Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013254.xml 2022-11-23T01:34:53.8613317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8613441Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8613720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8613852Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8614034Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph4_asgc7 2022-11-23T01:34:53.8614214Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph4_asgc7/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8614481Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8614488Z 2022-11-23T01:34:53.8614559Z Running tests... 2022-11-23T01:34:53.8614754Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8615016Z test_device_maps_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 921 2022-11-23T01:34:53.8615168Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 922 2022-11-23T01:34:53.8615317Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 923 2022-11-23T01:34:53.8615465Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 924 2022-11-23T01:34:53.8615725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8615848Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8616128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8616266Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8616452Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsirl6zh7 2022-11-23T01:34:53.8616652Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsirl6zh7/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8616924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8617046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8617325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8617449Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8617635Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt1cpdpon 2022-11-23T01:34:53.8617832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt1cpdpon/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8618101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8618271Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8618556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8618689Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8618872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxl96p5xs 2022-11-23T01:34:53.8619052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxl96p5xs/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8619322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8619475Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8619755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8619890Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8620077Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfblpr649 2022-11-23T01:34:53.8620271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfblpr649/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8620434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8620598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8620745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8620906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8621011Z skip: Need at least 2 CUDA devices (2.179s) 2022-11-23T01:34:53.8621017Z 2022-11-23T01:34:53.8621212Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8621286Z Ran 1 test in 2.179s 2022-11-23T01:34:53.8621291Z 2022-11-23T01:34:53.8621363Z OK (skipped=1) 2022-11-23T01:34:53.8621368Z 2022-11-23T01:34:53.8621450Z Generating XML reports... 2022-11-23T01:34:53.8621867Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013258.xml 2022-11-23T01:34:53.8622140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8622250Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8622530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8622666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8622849Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeotcdc9c 2022-11-23T01:34:53.8623046Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeotcdc9c/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8623312Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8623317Z 2022-11-23T01:34:53.8623388Z Running tests... 2022-11-23T01:34:53.8623583Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8623841Z test_device_maps_return_to_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 976 2022-11-23T01:34:53.8623993Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 977 2022-11-23T01:34:53.8624141Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 978 2022-11-23T01:34:53.8624294Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 979 2022-11-23T01:34:53.8624567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8624720Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8625002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8625137Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8625323Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq8nu7xti 2022-11-23T01:34:53.8625505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq8nu7xti/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8625778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8625932Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8626210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8626342Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8649114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgau7cjp4 2022-11-23T01:34:53.8649512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8649631Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8649915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8650043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8650224Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3rn_v_bn 2022-11-23T01:34:53.8650420Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgau7cjp4/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8650602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3rn_v_bn/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8650871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8650988Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8651264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8651392Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8651568Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps1zvi5fj 2022-11-23T01:34:53.8651754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps1zvi5fj/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8651907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8652062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8652218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8652371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8652466Z skip: Need at least 4 CUDA devices (2.181s) 2022-11-23T01:34:53.8652474Z 2022-11-23T01:34:53.8652660Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8652732Z Ran 1 test in 2.182s 2022-11-23T01:34:53.8652737Z 2022-11-23T01:34:53.8652809Z OK (skipped=1) 2022-11-23T01:34:53.8652814Z 2022-11-23T01:34:53.8652889Z Generating XML reports... 2022-11-23T01:34:53.8653297Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013302.xml 2022-11-23T01:34:53.8653566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8653683Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8654089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8654229Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8654415Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_6u29yf9 2022-11-23T01:34:53.8654603Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_6u29yf9/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8654868Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8654874Z 2022-11-23T01:34:53.8654983Z Running tests... 2022-11-23T01:34:53.8655169Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8655441Z test_device_maps_return_to_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1031 2022-11-23T01:34:53.8655594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1032 2022-11-23T01:34:53.8655743Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1033 2022-11-23T01:34:53.8655891Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1034 2022-11-23T01:34:53.8656161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8656279Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8656560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8656686Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8656867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5xo1t9xm 2022-11-23T01:34:53.8657060Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5xo1t9xm/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8657331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8657452Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8657728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8657857Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8658034Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmv1712uv 2022-11-23T01:34:53.8658222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmv1712uv/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8658482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8658603Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8658884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8659018Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8659201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdhn1xh2r 2022-11-23T01:34:53.8659396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdhn1xh2r/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8659668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8659793Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8660064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8660197Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8660443Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprzjzw1az 2022-11-23T01:34:53.8660638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprzjzw1az/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8660801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8660965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8661127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8661290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8661422Z skip: Need at least 4 CUDA devices (2.179s) 2022-11-23T01:34:53.8661428Z 2022-11-23T01:34:53.8661613Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8661690Z Ran 1 test in 2.179s 2022-11-23T01:34:53.8661696Z 2022-11-23T01:34:53.8661766Z OK (skipped=1) 2022-11-23T01:34:53.8661772Z 2022-11-23T01:34:53.8661858Z Generating XML reports... 2022-11-23T01:34:53.8662276Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013306.xml 2022-11-23T01:34:53.8662548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8662672Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8662954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8663090Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8663263Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmporzb8jb3 2022-11-23T01:34:53.8663460Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmporzb8jb3/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8663732Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8663737Z 2022-11-23T01:34:53.8663812Z Running tests... 2022-11-23T01:34:53.8664006Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8664284Z test_device_maps_wrong_worker_name (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1086 2022-11-23T01:34:53.8664440Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1087 2022-11-23T01:34:53.8664591Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1088 2022-11-23T01:34:53.8664732Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1089 2022-11-23T01:34:53.8665006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8665130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8665414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8665549Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8665734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkiadby4a 2022-11-23T01:34:53.8665929Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkiadby4a/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8666199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8666322Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8666591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8666729Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8666943Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe44iehr_ 2022-11-23T01:34:53.8667139Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe44iehr_/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8667408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8667533Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8667814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8667947Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8668164Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpocn8y3hn 2022-11-23T01:34:53.8668344Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpocn8y3hn/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8668616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8668739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8669020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8669154Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8669337Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_adhpnvq 2022-11-23T01:34:53.8669530Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_adhpnvq/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8669698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8669849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8670008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8670172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8670276Z skip: Need at least 2 CUDA devices (2.279s) 2022-11-23T01:34:53.8670282Z 2022-11-23T01:34:53.8670476Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8670552Z Ran 1 test in 2.279s 2022-11-23T01:34:53.8670557Z 2022-11-23T01:34:53.8670630Z OK (skipped=1) 2022-11-23T01:34:53.8670635Z 2022-11-23T01:34:53.8670718Z Generating XML reports... 2022-11-23T01:34:53.8671135Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013310.xml 2022-11-23T01:34:53.8671399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8671523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8671805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8671941Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8672126Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsgy685ri 2022-11-23T01:34:53.8672401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsgy685ri/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8672674Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8672679Z 2022-11-23T01:34:53.8672752Z Running tests... 2022-11-23T01:34:53.8672933Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8673200Z test_device_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1141 2022-11-23T01:34:53.8673356Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1142 2022-11-23T01:34:53.8673543Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1143 2022-11-23T01:34:53.8673698Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1144 2022-11-23T01:34:53.8673973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8674095Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8674377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8674514Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8674719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvpmph6u9 2022-11-23T01:34:53.8674917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvpmph6u9/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8675192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8675317Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8675599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8675735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8675921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm6bkn49i 2022-11-23T01:34:53.8676116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm6bkn49i/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8676390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8676500Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8676780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8676914Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8677097Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1e3gltrt 2022-11-23T01:34:53.8677295Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1e3gltrt/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8677563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8677686Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8677967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8678090Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8678273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvb6ktn2v 2022-11-23T01:34:53.8678471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvb6ktn2v/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8678635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8678798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8678958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8679119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8679219Z skip: Need at least 1 CUDA device (2.279s) 2022-11-23T01:34:53.8679228Z 2022-11-23T01:34:53.8679424Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8679486Z Ran 1 test in 2.279s 2022-11-23T01:34:53.8679491Z 2022-11-23T01:34:53.8679561Z OK (skipped=1) 2022-11-23T01:34:53.8679565Z 2022-11-23T01:34:53.8679650Z Generating XML reports... 2022-11-23T01:34:53.8680108Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013315.xml 2022-11-23T01:34:53.8680382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8680507Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8680905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8681043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8681216Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbnx8id_x 2022-11-23T01:34:53.8681462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbnx8id_x/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8681733Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8681741Z 2022-11-23T01:34:53.8681815Z Running tests... 2022-11-23T01:34:53.8682014Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8682284Z test_devices_option_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1196 2022-11-23T01:34:53.8682440Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1197 2022-11-23T01:34:53.8682596Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1198 2022-11-23T01:34:53.8682748Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1199 2022-11-23T01:34:53.8683012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8683138Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8683423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8683561Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8683748Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp349mbj6l 2022-11-23T01:34:53.8683943Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp349mbj6l/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8684216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8684340Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8684621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8684743Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8684930Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk0vqbiti 2022-11-23T01:34:53.8685127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk0vqbiti/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8685397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8685518Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8685799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8685934Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8686120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppstjujan 2022-11-23T01:34:53.8686305Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppstjujan/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8686577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8686751Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8687032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8687165Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8687353Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgmwiw_17 2022-11-23T01:34:53.8687546Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgmwiw_17/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8687710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8687903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8688053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8688214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8688318Z skip: Need at least 2 CUDA devices (2.181s) 2022-11-23T01:34:53.8688324Z 2022-11-23T01:34:53.8688523Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8688598Z Ran 1 test in 2.182s 2022-11-23T01:34:53.8688603Z 2022-11-23T01:34:53.8688673Z OK (skipped=1) 2022-11-23T01:34:53.8688677Z 2022-11-23T01:34:53.8688762Z Generating XML reports... 2022-11-23T01:34:53.8689180Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013319.xml 2022-11-23T01:34:53.8689457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8689571Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8689852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8689988Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8690174Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnlxpbdrg 2022-11-23T01:34:53.8690371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnlxpbdrg/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8690639Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8690644Z 2022-11-23T01:34:53.8690717Z Running tests... 2022-11-23T01:34:53.8690912Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8691177Z test_devices_option_mismatch_reverse (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1251 2022-11-23T01:34:53.8691332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1252 2022-11-23T01:34:53.8691488Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1253 2022-11-23T01:34:53.8691641Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1254 2022-11-23T01:34:53.8691914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8692036Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8692318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8692454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8692640Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj_0230sc 2022-11-23T01:34:53.8692821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj_0230sc/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8693123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8693249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8693531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8693665Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8693850Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb92_veuc 2022-11-23T01:34:53.8694044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb92_veuc/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8694313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8694453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8694734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8694873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8695055Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppr_6w5iq 2022-11-23T01:34:53.8695252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppr_6w5iq/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8695521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8695644Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8695921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8696057Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8696227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz_pwckra 2022-11-23T01:34:53.8696424Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz_pwckra/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8696589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8696750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8696910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8697072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8697177Z skip: Need at least 2 CUDA devices (2.282s) 2022-11-23T01:34:53.8697182Z 2022-11-23T01:34:53.8697379Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8697443Z Ran 1 test in 2.282s 2022-11-23T01:34:53.8697460Z 2022-11-23T01:34:53.8697519Z OK (skipped=1) 2022-11-23T01:34:53.8697523Z 2022-11-23T01:34:53.8697607Z Generating XML reports... 2022-11-23T01:34:53.8698030Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013323.xml 2022-11-23T01:34:53.8698303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8698428Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8698707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8698842Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8699028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp204ubwxn 2022-11-23T01:34:53.8699212Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp204ubwxn/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8699478Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8699484Z 2022-11-23T01:34:53.8699557Z Running tests... 2022-11-23T01:34:53.8699777Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8700061Z test_owner_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1306 2022-11-23T01:34:53.8700215Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1307 2022-11-23T01:34:53.8700367Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1308 2022-11-23T01:34:53.8700519Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1309 2022-11-23T01:34:53.8700792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8700935Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8701219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8701356Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8701541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe5701fo6 2022-11-23T01:34:53.8701733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe5701fo6/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8702003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8702125Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8702403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8702539Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8702711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpka4dx1_9 2022-11-23T01:34:53.8702909Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpka4dx1_9/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8703179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8703302Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8703581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8703716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8703899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy8yr2ys0 2022-11-23T01:34:53.8704094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy8yr2ys0/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8704351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8704475Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8704755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8704888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8705072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprt938kdw 2022-11-23T01:34:53.8705266Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprt938kdw/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8705431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8705594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8705757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8705907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8706035Z skip: Need at least 1 CUDA device (2.180s) 2022-11-23T01:34:53.8706041Z 2022-11-23T01:34:53.8706239Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8706315Z Ran 1 test in 2.180s 2022-11-23T01:34:53.8706320Z 2022-11-23T01:34:53.8706391Z OK (skipped=1) 2022-11-23T01:34:53.8706395Z 2022-11-23T01:34:53.8706479Z Generating XML reports... 2022-11-23T01:34:53.8706895Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013327.xml 2022-11-23T01:34:53.8707171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8707311Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8707590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8707727Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8707913Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplx5krops 2022-11-23T01:34:53.8708110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplx5krops/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8708378Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8708383Z 2022-11-23T01:34:53.8708455Z Running tests... 2022-11-23T01:34:53.8708650Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8708928Z test_owner_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1361 2022-11-23T01:34:53.8709073Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1362 2022-11-23T01:34:53.8709225Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1363 2022-11-23T01:34:53.8709380Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1364 2022-11-23T01:34:53.8709651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8709773Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8710052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8710188Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8710374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2mrzy7em 2022-11-23T01:34:53.8710571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2mrzy7em/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8710831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8710954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8711233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8711369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8711554Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprr82a534 2022-11-23T01:34:53.8711746Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprr82a534/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8712015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8712140Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8712474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8712653Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8712839Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ejp01j_ 2022-11-23T01:34:53.8713031Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ejp01j_/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8713307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8713429Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8713707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8713868Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8714052Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphcyvsbrz 2022-11-23T01:34:53.8714234Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphcyvsbrz/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8714398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8714562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8714723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8714885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8714988Z skip: Need at least 2 CUDA devices (2.278s) 2022-11-23T01:34:53.8714992Z 2022-11-23T01:34:53.8715188Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8715264Z Ran 1 test in 2.278s 2022-11-23T01:34:53.8715269Z 2022-11-23T01:34:53.8715328Z OK (skipped=1) 2022-11-23T01:34:53.8715342Z 2022-11-23T01:34:53.8715413Z Generating XML reports... 2022-11-23T01:34:53.8715832Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013331.xml 2022-11-23T01:34:53.8716109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8716234Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8716515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8716647Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8716828Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_dbe1qbz 2022-11-23T01:34:53.8717020Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_dbe1qbz/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8717273Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8717289Z 2022-11-23T01:34:53.8717348Z Running tests... 2022-11-23T01:34:53.8717544Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8717823Z test_owner_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1416 2022-11-23T01:34:53.8717979Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1417 2022-11-23T01:34:53.8718131Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1418 2022-11-23T01:34:53.8718285Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1419 2022-11-23T01:34:53.8718559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8718685Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8718954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8719119Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8719306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbc_xz84f 2022-11-23T01:34:53.8719499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbc_xz84f/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8719770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8719894Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8720174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8720338Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8720508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6e2zw4fb 2022-11-23T01:34:53.8720820Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6e2zw4fb/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8721092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8721214Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8721494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8721628Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8721812Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr7ssbf26 2022-11-23T01:34:53.8722004Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr7ssbf26/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8722276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8722387Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8722667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8722802Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8722986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0f6aek_r 2022-11-23T01:34:53.8723181Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0f6aek_r/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8723347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8723509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8723673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8723834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8723925Z skip: Need at least 2 CUDA devices (2.281s) 2022-11-23T01:34:53.8723933Z 2022-11-23T01:34:53.8724129Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8724204Z Ran 1 test in 2.281s 2022-11-23T01:34:53.8724210Z 2022-11-23T01:34:53.8724282Z OK (skipped=1) 2022-11-23T01:34:53.8724286Z 2022-11-23T01:34:53.8724371Z Generating XML reports... 2022-11-23T01:34:53.8724786Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013335.xml 2022-11-23T01:34:53.8725061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8725188Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8725455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8725589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8725822Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvgsmdbll 2022-11-23T01:34:53.8726022Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvgsmdbll/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8726289Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8726294Z 2022-11-23T01:34:53.8726367Z Running tests... 2022-11-23T01:34:53.8726561Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8726840Z test_owner_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1471 2022-11-23T01:34:53.8727033Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1472 2022-11-23T01:34:53.8727172Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1473 2022-11-23T01:34:53.8727328Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1474 2022-11-23T01:34:53.8727605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8727729Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8728011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8728148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8728332Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc9vgouxz 2022-11-23T01:34:53.8728529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc9vgouxz/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8728798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8728907Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8729187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8729323Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8729505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjtdx9r6y 2022-11-23T01:34:53.8729700Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjtdx9r6y/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8729971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8730096Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8730373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8730493Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8730678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptjjgw8fg 2022-11-23T01:34:53.8730873Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptjjgw8fg/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8731142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8731264Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8731541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8731676Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8731861Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqf56qy7k 2022-11-23T01:34:53.8732055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqf56qy7k/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8732244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8732408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8732570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8732731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8732835Z skip: Need at least 2 CUDA devices (2.181s) 2022-11-23T01:34:53.8732841Z 2022-11-23T01:34:53.8733039Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8733116Z Ran 1 test in 2.181s 2022-11-23T01:34:53.8733154Z 2022-11-23T01:34:53.8733224Z OK (skipped=1) 2022-11-23T01:34:53.8733229Z 2022-11-23T01:34:53.8733300Z Generating XML reports... 2022-11-23T01:34:53.8733718Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013339.xml 2022-11-23T01:34:53.8733995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8734118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8734397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8734533Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8734717Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfjner8mq 2022-11-23T01:34:53.8734915Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfjner8mq/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8735185Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8735190Z 2022-11-23T01:34:53.8735250Z Running tests... 2022-11-23T01:34:53.8735446Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8735727Z test_rref_as_arg_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1526 2022-11-23T01:34:53.8735886Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1527 2022-11-23T01:34:53.8736038Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1528 2022-11-23T01:34:53.8736189Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1529 2022-11-23T01:34:53.8736467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8736592Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8736871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8736993Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8737181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb3kfpjfk 2022-11-23T01:34:53.8737380Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb3kfpjfk/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8737651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8737774Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8738053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8738189Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8738374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4eby1xkn 2022-11-23T01:34:53.8738556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4eby1xkn/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8738857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8738979Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8739259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8739392Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8739575Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmp2hzpb4 2022-11-23T01:34:53.8739771Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmp2hzpb4/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8740069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8740192Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8740458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8740590Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8740775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmploklq65k 2022-11-23T01:34:53.8740967Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmploklq65k/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8741128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8741290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8741454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8741615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8741702Z skip: Need at least 1 CUDA device (2.182s) 2022-11-23T01:34:53.8741720Z 2022-11-23T01:34:53.8741906Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8741981Z Ran 1 test in 2.182s 2022-11-23T01:34:53.8741986Z 2022-11-23T01:34:53.8742056Z OK (skipped=1) 2022-11-23T01:34:53.8742060Z 2022-11-23T01:34:53.8742144Z Generating XML reports... 2022-11-23T01:34:53.8742563Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013343.xml 2022-11-23T01:34:53.8742836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8742960Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8743246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8743368Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8743561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf48logaw 2022-11-23T01:34:53.8743758Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf48logaw/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8744025Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8744030Z 2022-11-23T01:34:53.8744105Z Running tests... 2022-11-23T01:34:53.8744300Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8744574Z test_rref_as_arg_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1581 2022-11-23T01:34:53.8744734Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1582 2022-11-23T01:34:53.8744888Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1583 2022-11-23T01:34:53.8745027Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1584 2022-11-23T01:34:53.8745334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8745462Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8745746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8745883Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8746069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx1cwpkmu 2022-11-23T01:34:53.8746264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx1cwpkmu/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8746567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8746678Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8746957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8747092Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8747277Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdwlgd98f 2022-11-23T01:34:53.8747474Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdwlgd98f/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8747745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8747869Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8748152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8748287Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8748460Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0qoudp18 2022-11-23T01:34:53.8748657Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0qoudp18/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8748927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8749051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8749333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8749467Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8749650Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ng7hfn2 2022-11-23T01:34:53.8749848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ng7hfn2/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8750000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8750165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8750327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8750488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8750592Z skip: Need at least 2 CUDA devices (2.279s) 2022-11-23T01:34:53.8750597Z 2022-11-23T01:34:53.8750795Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8750871Z Ran 1 test in 2.279s 2022-11-23T01:34:53.8750876Z 2022-11-23T01:34:53.8750947Z OK (skipped=1) 2022-11-23T01:34:53.8750954Z 2022-11-23T01:34:53.8751037Z Generating XML reports... 2022-11-23T01:34:53.8751439Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013347.xml 2022-11-23T01:34:53.8751744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8751869Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8752147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8752281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8752542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphp8xkve_ 2022-11-23T01:34:53.8752738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphp8xkve_/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8753043Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8753048Z 2022-11-23T01:34:53.8753120Z Running tests... 2022-11-23T01:34:53.8753303Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8754062Z test_rref_as_arg_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81962 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.256s) 2022-11-23T01:34:53.8754083Z 2022-11-23T01:34:53.8754265Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8754341Z Ran 1 test in 0.256s 2022-11-23T01:34:53.8754346Z 2022-11-23T01:34:53.8754418Z OK (skipped=1) 2022-11-23T01:34:53.8754423Z 2022-11-23T01:34:53.8754510Z Generating XML reports... 2022-11-23T01:34:53.8754924Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013351.xml 2022-11-23T01:34:53.8755196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8755322Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8755601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8755723Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8755905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb_bk2jsa 2022-11-23T01:34:53.8756101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb_bk2jsa/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8756366Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8756373Z 2022-11-23T01:34:53.8756445Z Running tests... 2022-11-23T01:34:53.8756639Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8756916Z test_rref_as_arg_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1646 2022-11-23T01:34:53.8757072Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1647 2022-11-23T01:34:53.8757229Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1648 2022-11-23T01:34:53.8757369Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1649 2022-11-23T01:34:53.8757642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8757767Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8758048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8758186Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8758375Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpibbf3vnj 2022-11-23T01:34:53.8758599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpibbf3vnj/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8758873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8758982Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8759259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8759394Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8759578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptyf6kc2o 2022-11-23T01:34:53.8759833Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptyf6kc2o/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8760104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8760230Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8760513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8760781Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8760953Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp76slefqo 2022-11-23T01:34:53.8761148Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp76slefqo/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8761419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8761544Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8761823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8761961Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8762145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpttissj9e 2022-11-23T01:34:53.8762341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpttissj9e/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8762491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8762654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8762818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8762983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8763087Z skip: Need at least 2 CUDA devices (2.289s) 2022-11-23T01:34:53.8763092Z 2022-11-23T01:34:53.8763291Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8763367Z Ran 1 test in 2.289s 2022-11-23T01:34:53.8763373Z 2022-11-23T01:34:53.8763448Z OK (skipped=1) 2022-11-23T01:34:53.8763452Z 2022-11-23T01:34:53.8763536Z Generating XML reports... 2022-11-23T01:34:53.8763941Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013353.xml 2022-11-23T01:34:53.8764217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8764341Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8764622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8764760Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8764946Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnwcyuoaj 2022-11-23T01:34:53.8765192Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnwcyuoaj/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8765462Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8765467Z 2022-11-23T01:34:53.8765539Z Running tests... 2022-11-23T01:34:53.8765724Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8766001Z test_rref_as_arg_synchronization5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1701 2022-11-23T01:34:53.8766156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1702 2022-11-23T01:34:53.8766345Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1703 2022-11-23T01:34:53.8766496Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1704 2022-11-23T01:34:53.8766774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8766900Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8767185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8767307Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8767493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8o5tvcse 2022-11-23T01:34:53.8767691Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8o5tvcse/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8767962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8768086Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8768367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8768504Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8768685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkz6_vhmn 2022-11-23T01:34:53.8768880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkz6_vhmn/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8769137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8769259Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8769540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8769676Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8769858Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiv65diuc 2022-11-23T01:34:53.8770131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8770253Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8770532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8770666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8770836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx5oa272i 2022-11-23T01:34:53.8771031Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiv65diuc/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8771219Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx5oa272i/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8771386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8771550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8771738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8771900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8772001Z skip: Need at least 1 CUDA device (2.181s) 2022-11-23T01:34:53.8772007Z 2022-11-23T01:34:53.8772191Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8772268Z Ran 1 test in 2.181s 2022-11-23T01:34:53.8772273Z 2022-11-23T01:34:53.8772343Z OK (skipped=1) 2022-11-23T01:34:53.8772348Z 2022-11-23T01:34:53.8772433Z Generating XML reports... 2022-11-23T01:34:53.8773075Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013357.xml 2022-11-23T01:34:53.8773379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8773504Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8773784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8773919Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8774089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzqf7_jwq 2022-11-23T01:34:53.8774281Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzqf7_jwq/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8774547Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8774554Z 2022-11-23T01:34:53.8774626Z Running tests... 2022-11-23T01:34:53.8774822Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8775098Z test_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1756 2022-11-23T01:34:53.8775254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1757 2022-11-23T01:34:53.8775405Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1758 2022-11-23T01:34:53.8775559Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1759 2022-11-23T01:34:53.8775821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8775943Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8776220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8776357Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8776539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptryz7j78 2022-11-23T01:34:53.8776736Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptryz7j78/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8777007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8777129Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8777393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8777528Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8777708Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5tan4vle 2022-11-23T01:34:53.8777904Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5tan4vle/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8778175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8778299Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8778618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8778754Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8778937Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp96nqoct0 2022-11-23T01:34:53.8779117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp96nqoct0/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8779385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8779536Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8779814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8779947Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8780132Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp4f1_h23 2022-11-23T01:34:53.8780324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp4f1_h23/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8780488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8780650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8780798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8780959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8781062Z skip: Need at least 1 CUDA device (2.288s) 2022-11-23T01:34:53.8781068Z 2022-11-23T01:34:53.8781265Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8781339Z Ran 1 test in 2.288s 2022-11-23T01:34:53.8781345Z 2022-11-23T01:34:53.8781414Z OK (skipped=1) 2022-11-23T01:34:53.8781421Z 2022-11-23T01:34:53.8781505Z Generating XML reports... 2022-11-23T01:34:53.8781920Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013401.xml 2022-11-23T01:34:53.8782177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8782300Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8782580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8782717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8782900Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpegv9s_5t 2022-11-23T01:34:53.8783093Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpegv9s_5t/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8783361Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8783367Z 2022-11-23T01:34:53.8783439Z Running tests... 2022-11-23T01:34:53.8783632Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8783900Z test_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1811 2022-11-23T01:34:53.8784054Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1812 2022-11-23T01:34:53.8784205Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1813 2022-11-23T01:34:53.8784358Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1814 2022-11-23T01:34:53.8784634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8784803Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8785090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8785225Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8785409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpltsidade 2022-11-23T01:34:53.8785593Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpltsidade/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8785864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8786016Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8786294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8786427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8786615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy1zv03cs 2022-11-23T01:34:53.8786807Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy1zv03cs/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8787083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8787193Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8787472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8787606Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8787792Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfsigwjxe 2022-11-23T01:34:53.8787988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfsigwjxe/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8788258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8788380Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8788656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8788792Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8788961Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5v7edqer 2022-11-23T01:34:53.8789154Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5v7edqer/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8789319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8789480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8789642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8789801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8789906Z skip: Need at least 2 CUDA devices (2.180s) 2022-11-23T01:34:53.8789910Z 2022-11-23T01:34:53.8790103Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8790167Z Ran 1 test in 2.180s 2022-11-23T01:34:53.8790187Z 2022-11-23T01:34:53.8790244Z OK (skipped=1) 2022-11-23T01:34:53.8790249Z 2022-11-23T01:34:53.8790333Z Generating XML reports... 2022-11-23T01:34:53.8790750Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013406.xml 2022-11-23T01:34:53.8791025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8791148Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8791456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8791593Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8791779Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxklq1ecv 2022-11-23T01:34:53.8791961Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxklq1ecv/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8792229Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8792234Z 2022-11-23T01:34:53.8792382Z Running tests... 2022-11-23T01:34:53.8792628Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8792906Z test_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1866 2022-11-23T01:34:53.8793064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1867 2022-11-23T01:34:53.8793219Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1868 2022-11-23T01:34:53.8793372Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1869 2022-11-23T01:34:53.8793646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8793756Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8794039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8794178Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8794365Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4tsjbce4 2022-11-23T01:34:53.8794562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4tsjbce4/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8794840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8794965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8795249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8795371Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8795558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpinifpcit 2022-11-23T01:34:53.8795757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpinifpcit/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8796030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8796155Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8796439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8796571Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8796755Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxrb7zgmn 2022-11-23T01:34:53.8796948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxrb7zgmn/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8797205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8797327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8797610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8797744Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8797956Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8z_r723c 2022-11-23T01:34:53.8798149Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8z_r723c/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8798313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8798476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8798625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8798785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8798889Z skip: Need at least 2 CUDA devices (2.180s) 2022-11-23T01:34:53.8798923Z 2022-11-23T01:34:53.8799120Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8799199Z Ran 1 test in 2.180s 2022-11-23T01:34:53.8799204Z 2022-11-23T01:34:53.8799275Z OK (skipped=1) 2022-11-23T01:34:53.8799279Z 2022-11-23T01:34:53.8799363Z Generating XML reports... 2022-11-23T01:34:53.8799782Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013410.xml 2022-11-23T01:34:53.8800056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8800166Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8800446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8800671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8800858Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_tvpuhok 2022-11-23T01:34:53.8801052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_tvpuhok/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8801323Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8801328Z 2022-11-23T01:34:53.8801401Z Running tests... 2022-11-23T01:34:53.8801599Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8801876Z test_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1921 2022-11-23T01:34:53.8802019Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1922 2022-11-23T01:34:53.8802170Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1923 2022-11-23T01:34:53.8802324Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1924 2022-11-23T01:34:53.8802597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8802721Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8803003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8803138Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8803323Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6_mx8gay 2022-11-23T01:34:53.8803506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6_mx8gay/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8803778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8803901Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8804181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8804317Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8804547Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqwlkfyir 2022-11-23T01:34:53.8804746Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqwlkfyir/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8805016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8805138Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8805400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8805533Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8805754Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpugilbf1v 2022-11-23T01:34:53.8805949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpugilbf1v/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8806222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8806346Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8806628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8806763Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8806947Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvkmsgtyu 2022-11-23T01:34:53.8807128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvkmsgtyu/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8807293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8807456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8807617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8807779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8807886Z skip: Need at least 2 CUDA devices (2.283s) 2022-11-23T01:34:53.8807891Z 2022-11-23T01:34:53.8808085Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8808160Z Ran 1 test in 2.283s 2022-11-23T01:34:53.8808165Z 2022-11-23T01:34:53.8808223Z OK (skipped=1) 2022-11-23T01:34:53.8808228Z 2022-11-23T01:34:53.8808313Z Generating XML reports... 2022-11-23T01:34:53.8808730Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013414.xml 2022-11-23T01:34:53.8809005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8809128Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8809412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8809546Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8809729Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpulk7rvtu 2022-11-23T01:34:53.8809922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpulk7rvtu/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8810176Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8810181Z 2022-11-23T01:34:53.8810256Z Running tests... 2022-11-23T01:34:53.8810450Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8810728Z test_rref_to_here_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1976 2022-11-23T01:34:53.8810884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1977 2022-11-23T01:34:53.8811067Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1978 2022-11-23T01:34:53.8811221Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1979 2022-11-23T01:34:53.8811495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8811619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8811888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8812023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8812235Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnyl5s4uz 2022-11-23T01:34:53.8812428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnyl5s4uz/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8812704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8812826Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8813108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8813243Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8813416Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo4oifmp9 2022-11-23T01:34:53.8813610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo4oifmp9/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8813883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8814005Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8814283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8814418Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8814601Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpet2i864q 2022-11-23T01:34:53.8814795Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpet2i864q/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8815064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8815173Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8815452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8815587Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8815770Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7itvw6x2 2022-11-23T01:34:53.8815966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7itvw6x2/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8816131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8816293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8816454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8816601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8816702Z skip: Need at least 1 CUDA device (2.182s) 2022-11-23T01:34:53.8816707Z 2022-11-23T01:34:53.8816904Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8816980Z Ran 1 test in 2.183s 2022-11-23T01:34:53.8816985Z 2022-11-23T01:34:53.8817058Z OK (skipped=1) 2022-11-23T01:34:53.8817063Z 2022-11-23T01:34:53.8817145Z Generating XML reports... 2022-11-23T01:34:53.8817592Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013418.xml 2022-11-23T01:34:53.8817869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8817992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8818258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8818392Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8818576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6nh2sgsr 2022-11-23T01:34:53.8818799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6nh2sgsr/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8819066Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8819073Z 2022-11-23T01:34:53.8819144Z Running tests... 2022-11-23T01:34:53.8819336Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8819609Z test_rref_to_here_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2031 2022-11-23T01:34:53.8819763Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2032 2022-11-23T01:34:53.8819903Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2033 2022-11-23T01:34:53.8820056Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2034 2022-11-23T01:34:53.8820331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8820454Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8820738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8820871Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8821055Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7gf818u1 2022-11-23T01:34:53.8821250Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7gf818u1/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8821513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8821635Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8821911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8822046Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8822230Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp28epb_v3 2022-11-23T01:34:53.8822425Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp28epb_v3/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8822693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8822817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8823096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8823217Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8823401Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0xprca9v 2022-11-23T01:34:53.8823598Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0xprca9v/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8823866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8824028Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8824305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8824437Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8824622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpda0lrk_r 2022-11-23T01:34:53.8824808Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpda0lrk_r/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8824971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8825160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8825321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8825485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8825589Z skip: Need at least 2 CUDA devices (2.179s) 2022-11-23T01:34:53.8825595Z 2022-11-23T01:34:53.8825793Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8825867Z Ran 1 test in 2.179s 2022-11-23T01:34:53.8825873Z 2022-11-23T01:34:53.8825942Z OK (skipped=1) 2022-11-23T01:34:53.8825947Z 2022-11-23T01:34:53.8826018Z Generating XML reports... 2022-11-23T01:34:53.8826436Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013422.xml 2022-11-23T01:34:53.8826711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8826836Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8827114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8827251Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8827435Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphla0vsl9 2022-11-23T01:34:53.8827629Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphla0vsl9/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8827892Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8827897Z 2022-11-23T01:34:53.8827956Z Running tests... 2022-11-23T01:34:53.8828150Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8828431Z test_rref_to_here_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2086 2022-11-23T01:34:53.8828586Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2087 2022-11-23T01:34:53.8828741Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2088 2022-11-23T01:34:53.8828891Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2089 2022-11-23T01:34:53.8829164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8829286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8829553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8829688Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8829874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbmcpoo8z 2022-11-23T01:34:53.8830073Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbmcpoo8z/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8830377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8830501Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8830780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8830914Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8831095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzqqscon0 2022-11-23T01:34:53.8831277Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzqqscon0/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8831545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8831700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8831979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8832115Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8832374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpswqpx2e6 2022-11-23T01:34:53.8832574Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpswqpx2e6/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8832849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8832959Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8833238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8833376Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8833559Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf4blvklp 2022-11-23T01:34:53.8833756Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf4blvklp/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8833922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8834084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8834247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8834407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8834496Z skip: Need at least 2 CUDA devices (2.278s) 2022-11-23T01:34:53.8834502Z 2022-11-23T01:34:53.8834697Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8834773Z Ran 1 test in 2.278s 2022-11-23T01:34:53.8834779Z 2022-11-23T01:34:53.8834849Z OK (skipped=1) 2022-11-23T01:34:53.8834853Z 2022-11-23T01:34:53.8834941Z Generating XML reports... 2022-11-23T01:34:53.8835361Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013426.xml 2022-11-23T01:34:53.8835634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8835757Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8836038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8836161Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8836346Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7hhgri08 2022-11-23T01:34:53.8836544Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7hhgri08/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8836813Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8836818Z 2022-11-23T01:34:53.8836889Z Running tests... 2022-11-23T01:34:53.8837116Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8837392Z test_rref_to_here_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2141 2022-11-23T01:34:53.8837546Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2142 2022-11-23T01:34:53.8837687Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2143 2022-11-23T01:34:53.8837839Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2144 2022-11-23T01:34:53.8838114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8838269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8838551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8838687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8838875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptbgdt7w9 2022-11-23T01:34:53.8839074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptbgdt7w9/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8839347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8839456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8839736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8839877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8840059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp40t55mrr 2022-11-23T01:34:53.8840256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp40t55mrr/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8840527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8840770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8841057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8841191Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8841364Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn09wwri5 2022-11-23T01:34:53.8841559Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn09wwri5/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8841829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8841955Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8842236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8842371Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8842554Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5lme34sh 2022-11-23T01:34:53.8842748Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5lme34sh/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8842899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8843067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8843232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8843394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8843543Z skip: Need at least 2 CUDA devices (2.278s) 2022-11-23T01:34:53.8843550Z 2022-11-23T01:34:53.8843748Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8843825Z Ran 1 test in 2.279s 2022-11-23T01:34:53.8843830Z 2022-11-23T01:34:53.8843902Z OK (skipped=1) 2022-11-23T01:34:53.8843907Z 2022-11-23T01:34:53.8843991Z Generating XML reports... 2022-11-23T01:34:53.8844397Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013430.xml 2022-11-23T01:34:53.8844671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8844832Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8845113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8845251Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8845438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpktfy6j7b 2022-11-23T01:34:53.8845635Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpktfy6j7b/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8845902Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8845907Z 2022-11-23T01:34:53.8845978Z Running tests... 2022-11-23T01:34:53.8846162Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8846442Z test_rref_with_unpickleable_attributes (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2196 2022-11-23T01:34:53.8846599Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2197 2022-11-23T01:34:53.8846751Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2198 2022-11-23T01:34:53.8846905Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2199 2022-11-23T01:34:53.8847179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8847303Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8847585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8847707Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8847891Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5r8l3gnc 2022-11-23T01:34:53.8848088Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5r8l3gnc/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8848355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8848480Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8848762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8848894Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8849078Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnyp65qvw 2022-11-23T01:34:53.8849271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnyp65qvw/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8849527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8849650Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8849931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8850065Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8850286Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1wzhr58o 2022-11-23T01:34:53.8850481Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1wzhr58o/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8850750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8850873Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8851137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8851302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8851484Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw9q7eyg3 2022-11-23T01:34:53.8851675Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw9q7eyg3/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8851838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8852003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8852164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8852326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8852428Z skip: Need at least 1 CUDA device (2.283s) 2022-11-23T01:34:53.8852433Z 2022-11-23T01:34:53.8852617Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8852696Z Ran 1 test in 2.283s 2022-11-23T01:34:53.8852701Z 2022-11-23T01:34:53.8852772Z OK (skipped=1) 2022-11-23T01:34:53.8852777Z 2022-11-23T01:34:53.8852861Z Generating XML reports... 2022-11-23T01:34:53.8853282Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013434.xml 2022-11-23T01:34:53.8853556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8853679Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8853956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8854090Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8854260Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6z5bjjj7 2022-11-23T01:34:53.8854454Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6z5bjjj7/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8854722Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8854727Z 2022-11-23T01:34:53.8854799Z Running tests... 2022-11-23T01:34:53.8854994Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8855264Z test_tensor_view_as_return_value (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2251 2022-11-23T01:34:53.8855421Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2252 2022-11-23T01:34:53.8855574Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2253 2022-11-23T01:34:53.8855713Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2254 2022-11-23T01:34:53.8855990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8856114Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8856394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8856555Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8856739Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1x3v0fco 2022-11-23T01:34:53.8856935Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1x3v0fco/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8857206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8857329Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8857594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8857756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8857937Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0n4albx5 2022-11-23T01:34:53.8858208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8858330Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8858610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8858744Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8858930Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaaxfryfn 2022-11-23T01:34:53.8859112Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0n4albx5/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8859300Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaaxfryfn/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8859571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8859693Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8859973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8860107Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8860290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwwa766mk 2022-11-23T01:34:53.8860487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwwa766mk/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8860650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8860800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8860962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8861123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8861223Z skip: Need at least 1 CUDA device (2.182s) 2022-11-23T01:34:53.8861228Z 2022-11-23T01:34:53.8861425Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8861504Z Ran 1 test in 2.182s 2022-11-23T01:34:53.8861509Z 2022-11-23T01:34:53.8861579Z OK (skipped=1) 2022-11-23T01:34:53.8861584Z 2022-11-23T01:34:53.8861667Z Generating XML reports... 2022-11-23T01:34:53.8862072Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013438.xml 2022-11-23T01:34:53.8862345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8862469Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8862752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8862888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8863138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbrnzme2s 2022-11-23T01:34:53.8863335Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbrnzme2s/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8863604Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8863609Z 2022-11-23T01:34:53.8863681Z Running tests... 2022-11-23T01:34:53.8863865Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8864151Z test_device_maps_backward_pass (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2306 2022-11-23T01:34:53.8864335Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2307 2022-11-23T01:34:53.8864488Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2308 2022-11-23T01:34:53.8864645Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2309 2022-11-23T01:34:53.8864919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8865044Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8865326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8865460Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8865633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw3id89vs 2022-11-23T01:34:53.8865828Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw3id89vs/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8866105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8866230Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8866509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8866643Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8866944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpai1nq9y4 2022-11-23T01:34:53.8867140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpai1nq9y4/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8867410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8867519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8867803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8867937Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8868126Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb5z5dk0_ 2022-11-23T01:34:53.8868321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb5z5dk0_/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8868589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8868713Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8868991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8869112Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8869295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpttnav2e1 2022-11-23T01:34:53.8869488Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpttnav2e1/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8869698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8869862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8870022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8870182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8870286Z skip: Need at least 4 CUDA devices (2.279s) 2022-11-23T01:34:53.8870291Z 2022-11-23T01:34:53.8870488Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8870551Z Ran 1 test in 2.279s 2022-11-23T01:34:53.8870556Z 2022-11-23T01:34:53.8870655Z OK (skipped=1) 2022-11-23T01:34:53.8870659Z 2022-11-23T01:34:53.8870746Z Generating XML reports... 2022-11-23T01:34:53.8871181Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221123013442.xml 2022-11-23T01:34:53.8871457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8871585Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8871863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8871997Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8872169Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwz6vv8h0 2022-11-23T01:34:53.8872435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwz6vv8h0/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8872713Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8872718Z 2022-11-23T01:34:53.8872792Z Running tests... 2022-11-23T01:34:53.8872990Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8873272Z test_dist_autograd_sync_streams (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2361 2022-11-23T01:34:53.8873430Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2362 2022-11-23T01:34:53.8873585Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2363 2022-11-23T01:34:53.8873735Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2364 2022-11-23T01:34:53.8873998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8874124Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8874406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8874543Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8874734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi50paghv 2022-11-23T01:34:53.8874928Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi50paghv/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8875201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8875322Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8875602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8875723Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8875909Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2z22261e 2022-11-23T01:34:53.8876101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2z22261e/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8876409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8876533Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8876812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8876945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8877125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkl_mz4dm 2022-11-23T01:34:53.8877306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkl_mz4dm/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8877602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8877723Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8878008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8878141Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8878323Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf18v3f63 2022-11-23T01:34:53.8878516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf18v3f63/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8878680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8878843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8878993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8879154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8879258Z skip: Need at least 4 CUDA devices (2.282s) 2022-11-23T01:34:53.8879263Z 2022-11-23T01:34:53.8879459Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8879533Z Ran 1 test in 2.282s 2022-11-23T01:34:53.8879538Z 2022-11-23T01:34:53.8879608Z OK (skipped=1) 2022-11-23T01:34:53.8879612Z 2022-11-23T01:34:53.8879697Z Generating XML reports... 2022-11-23T01:34:53.8880132Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221123013446.xml 2022-11-23T01:34:53.8880403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8880512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8880901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8881039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8881230Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpouaav9ig 2022-11-23T01:34:53.8881427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpouaav9ig/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8881694Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-11-23T01:34:53.8881700Z 2022-11-23T01:34:53.8881775Z Running tests... 2022-11-23T01:34:53.8881968Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8882238Z test_gradients_synchronizations (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2416 2022-11-23T01:34:53.8882396Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2417 2022-11-23T01:34:53.8882549Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2418 2022-11-23T01:34:53.8882701Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2419 2022-11-23T01:34:53.8883023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8883149Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8883431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8883568Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8883751Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp073vwj7j 2022-11-23T01:34:53.8883932Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp073vwj7j/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8884244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8884365Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8884650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8884788Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8884974Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4l1rdxzh 2022-11-23T01:34:53.8885169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4l1rdxzh/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8885439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8885548Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8885831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8885964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8886148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjtnf1_ar 2022-11-23T01:34:53.8886344Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjtnf1_ar/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8886614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:34:53.8886736Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:34:53.8887012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:34:53.8887145Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:34:53.8887316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqyt3sga3 2022-11-23T01:34:53.8887511Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqyt3sga3/_remote_module_non_scriptable.py 2022-11-23T01:34:53.8887675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:34:53.8887839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:34:53.8888000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:34:53.8888156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:34:53.8888260Z skip: Need at least 4 CUDA devices (2.180s) 2022-11-23T01:34:53.8888265Z 2022-11-23T01:34:53.8888461Z ---------------------------------------------------------------------- 2022-11-23T01:34:53.8888522Z Ran 1 test in 2.180s 2022-11-23T01:34:53.8888541Z 2022-11-23T01:34:53.8888599Z OK (skipped=1) 2022-11-23T01:34:53.8888606Z 2022-11-23T01:34:53.8888690Z Generating XML reports... 2022-11-23T01:34:53.8889125Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221123013450.xml 2022-11-23T01:34:53.8889130Z 2022-11-23T01:34:53.8889530Z ##[endgroup] 2022-11-23T01:34:53.8889959Z FINISHED PRINTING LOG FILE of distributed/rpc/cuda/test_tensorpipe_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-cuda-test_tensorpipe_agent_myzf8yej) 2022-11-23T01:34:53.8889966Z 2022-11-23T01:34:53.8890150Z Running distributed/test_c10d_gloo ... [2022-11-23 01:34:53.746005] 2022-11-23T01:34:53.8890523Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_gloo.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:34:53.746506] 2022-11-23T01:41:59.9506826Z 2022-11-23T01:41:59.9507197Z Expand the folded group to see the log file of distributed/test_c10d_gloo 2022-11-23T01:41:59.9507915Z ##[group]PRINTING LOG FILE of distributed/test_c10d_gloo (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_gloo_j6rogl31) 2022-11-23T01:41:59.9512016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphjcmckgr 2022-11-23T01:41:59.9512758Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphjcmckgr/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9514311Z , <__main__.CommTest testMethod=test_broadcast_coalesced_gloo_cuda>, <__main__.CommTest testMethod=test_gloo_barrier_device_ids>, <__main__.CommTest testMethod=test_gloo_rank_membership>, <__main__.CommTest testMethod=test_gloo_warn_not_in_group>, <__main__.CommTest testMethod=test_sequence_num_incremented_gloo_default>, <__main__.CommTest testMethod=test_sequence_num_incremented_gloo_subgroup>, <__main__.CommTest testMethod=test_sequence_num_set_default_pg_gloo>, <__main__.CommTest testMethod=test_sequence_num_set_gloo_new_group>, <__main__.CommTest testMethod=test_tensor_dtype_complex>, <__main__.CommTest testMethod=test_tensor_dtype_mismatch>]> 2022-11-23T01:41:59.9515878Z test_broadcast_coalesced_gloo_cpu (__main__.CommTest) 2022-11-23T01:41:59.9516351Z test_broadcast_coalesced_gloo_cuda (__main__.CommTest) 2022-11-23T01:41:59.9516759Z test_gloo_barrier_device_ids (__main__.CommTest) 2022-11-23T01:41:59.9517364Z test_gloo_rank_membership (__main__.CommTest) 2022-11-23T01:41:59.9517746Z test_gloo_warn_not_in_group (__main__.CommTest) 2022-11-23T01:41:59.9518180Z test_sequence_num_incremented_gloo_default (__main__.CommTest) 2022-11-23T01:41:59.9518666Z test_sequence_num_incremented_gloo_subgroup (__main__.CommTest) 2022-11-23T01:41:59.9519000Z test_sequence_num_set_default_pg_gloo (__main__.CommTest) 2022-11-23T01:41:59.9519392Z test_sequence_num_set_gloo_new_group (__main__.CommTest) 2022-11-23T01:41:59.9519834Z test_tensor_dtype_complex (__main__.CommTest) 2022-11-23T01:41:59.9520266Z test_tensor_dtype_mismatch (__main__.CommTest) 2022-11-23T01:41:59.9522170Z , <__main__.CompilerTest testMethod=test_allgather_work_wait_gpu>, <__main__.CompilerTest testMethod=test_allreduce_work_wait_cpu>, <__main__.CompilerTest testMethod=test_allreduce_work_wait_gpu>, <__main__.CompilerTest testMethod=test_broadcast_work_wait_cpu>, <__main__.CompilerTest testMethod=test_broadcast_work_wait_gpu>, <__main__.CompilerTest testMethod=test_consecutive_comm_work_wait_cpu>, <__main__.CompilerTest testMethod=test_consecutive_comm_work_wait_gpu>, <__main__.CompilerTest testMethod=test_nested_comm_tensor_wrapping>, <__main__.CompilerTest testMethod=test_scatter_work_wait_cpu>, <__main__.CompilerTest testMethod=test_scatter_work_wait_gpu>]> 2022-11-23T01:41:59.9523685Z test_allgather_work_wait_cpu (__main__.CompilerTest) 2022-11-23T01:41:59.9524094Z test_allgather_work_wait_gpu (__main__.CompilerTest) 2022-11-23T01:41:59.9524522Z test_allreduce_work_wait_cpu (__main__.CompilerTest) 2022-11-23T01:41:59.9524933Z test_allreduce_work_wait_gpu (__main__.CompilerTest) 2022-11-23T01:41:59.9525348Z test_broadcast_work_wait_cpu (__main__.CompilerTest) 2022-11-23T01:41:59.9525777Z test_broadcast_work_wait_gpu (__main__.CompilerTest) 2022-11-23T01:41:59.9526396Z test_consecutive_comm_work_wait_cpu (__main__.CompilerTest) 2022-11-23T01:41:59.9526851Z test_consecutive_comm_work_wait_gpu (__main__.CompilerTest) 2022-11-23T01:41:59.9527290Z test_nested_comm_tensor_wrapping (__main__.CompilerTest) 2022-11-23T01:41:59.9527720Z test_scatter_work_wait_cpu (__main__.CompilerTest) 2022-11-23T01:41:59.9528125Z test_scatter_work_wait_gpu (__main__.CompilerTest) 2022-11-23T01:41:59.9535736Z , <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_cpu>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_gpu_gloo>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_register_just_once>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_sparse_gradients>, <__main__.DistributedDataParallelTest testMethod=test_ddp_invalid_comm_hook_init>, <__main__.DistributedDataParallelTest testMethod=test_ddp_invalid_comm_hook_return_type>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_when_unused_parameters_empty>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad_with_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad_with_static_graph>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_1gpu_module_device_ids_integer_list>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_1gpu_module_device_ids_torch_device_list>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_2gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_4gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_cpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_cpu_module_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ignored_output>, <__main__.DistributedDataParallelTest testMethod=test_ignored_output_with_unused_parameters>, <__main__.DistributedDataParallelTest testMethod=test_ignored_sharded_tensor>, <__main__.DistributedDataParallelTest testMethod=test_invalid_powerSGD_state>, <__main__.DistributedDataParallelTest testMethod=test_save_load_checkpoint>, <__main__.DistributedDataParallelTest testMethod=test_sparse_gradients>, <__main__.DistributedDataParallelTest testMethod=test_sparse_gradients_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_empty_input>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_only_empty_input>]> 2022-11-23T01:41:59.9543611Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9544283Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9544881Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9545500Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9546160Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9546808Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9547423Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9548033Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9548624Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9549224Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9549843Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9550520Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9551210Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9551858Z test_ddp_comm_hook_future_passing_cpu (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9552461Z test_ddp_comm_hook_future_passing_gpu_gloo (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9553118Z test_ddp_comm_hook_register_just_once (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9553722Z test_ddp_comm_hook_sparse_gradients (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9554307Z test_ddp_invalid_comm_hook_init (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9554882Z test_ddp_invalid_comm_hook_return_type (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9555467Z test_find_unused_parameters_when_unused_parameters_empty (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9556050Z test_global_local_unused_params_grad (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9556596Z test_global_local_unused_params_grad_with_grad_is_view (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9557191Z test_global_local_unused_params_grad_with_static_graph (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9557786Z test_gloo_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9558396Z test_gloo_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9558956Z test_gloo_backend_2gpu_module (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9559488Z test_gloo_backend_4gpu_module (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9559999Z test_gloo_backend_cpu_module (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9560531Z test_gloo_backend_cpu_module_grad_is_view (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9561188Z test_ignored_output (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9561773Z test_ignored_output_with_unused_parameters (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9562371Z test_ignored_sharded_tensor (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9562908Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9563435Z test_save_load_checkpoint (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9563937Z test_sparse_gradients (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9564459Z test_sparse_gradients_grad_is_view (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9564969Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9565528Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9567087Z , <__main__.GlooProcessGroupWithDispatchedCollectivesTests testMethod=test_allreduce_coalesced>, <__main__.GlooProcessGroupWithDispatchedCollectivesTests testMethod=test_collectives>, <__main__.GlooProcessGroupWithDispatchedCollectivesTests testMethod=test_monitored_barrier>]> 2022-11-23T01:41:59.9568588Z test_allgather_coalesced (__main__.GlooProcessGroupWithDispatchedCollectivesTests) 2022-11-23T01:41:59.9569321Z test_allreduce_coalesced (__main__.GlooProcessGroupWithDispatchedCollectivesTests) 2022-11-23T01:41:59.9570102Z test_collectives (__main__.GlooProcessGroupWithDispatchedCollectivesTests) 2022-11-23T01:41:59.9570763Z test_monitored_barrier (__main__.GlooProcessGroupWithDispatchedCollectivesTests) 2022-11-23T01:41:59.9571305Z 2022-11-23T01:41:59.9578589Z , <__main__.ProcessGroupGlooTest testMethod=test_allgather_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_coalesced_async>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_coalesced_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_noncontiguous_input>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_cuda_using_work_api>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_using_work_api>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_async>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_checks_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_barrier_implies_wait>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_basics>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_checks>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_stress>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_empty_tensors>, <__main__.ProcessGroupGlooTest testMethod=test_gather_basics>, <__main__.ProcessGroupGlooTest testMethod=test_gather_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_gather_checks>, <__main__.ProcessGroupGlooTest testMethod=test_gather_noncontiguous_input>, <__main__.ProcessGroupGlooTest testMethod=test_gather_stress>, <__main__.ProcessGroupGlooTest testMethod=test_gather_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_multi_device_constructor>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_checks>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_stress>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_round_robin>, <__main__.ProcessGroupGlooTest testMethod=test_round_robin_create_destroy>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_basics>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_checks>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_stress>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_send_recv_all_to_all>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_checks>]> 2022-11-23T01:41:59.9584807Z test_allgather_basics (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9585097Z test_allgather_basics_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9585374Z test_allgather_checks (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9585677Z test_allgather_coalesced_async (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9585969Z test_allgather_coalesced_checks (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9586270Z test_allgather_noncontiguous_input (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9586558Z test_allgather_stress (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9586824Z test_allgather_stress_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9587097Z test_allreduce_basics (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9587371Z test_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9587656Z test_allreduce_basics_cuda_using_work_api (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9587965Z test_allreduce_basics_using_work_api (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9588249Z test_allreduce_checks (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9588516Z test_allreduce_coalesced_async (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9588805Z test_allreduce_coalesced_basics (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9589096Z test_allreduce_coalesced_checks (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9589396Z test_allreduce_coalesced_checks_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9589683Z test_allreduce_coalesced_stress (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9589962Z test_allreduce_stress (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9590235Z test_allreduce_stress_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9590503Z test_barrier_implies_wait (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9590771Z test_broadcast_basics (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9591041Z test_broadcast_basics_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9591296Z test_broadcast_checks (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9591564Z test_broadcast_stress (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9591833Z test_broadcast_stress_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9592090Z test_empty_tensors (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9592348Z test_gather_basics (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9592687Z test_gather_basics_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9592954Z test_gather_checks (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9593223Z test_gather_noncontiguous_input (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9593500Z test_gather_stress (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9593768Z test_gather_stress_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9594040Z test_multi_device_constructor (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9594311Z test_reduce_basics (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9594575Z test_reduce_basics_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9594827Z test_reduce_checks (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9595082Z test_reduce_stress (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9595344Z test_reduce_stress_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9595609Z test_round_robin (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9595867Z test_round_robin_create_destroy (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9596138Z test_scatter_basics (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9596450Z test_scatter_basics_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9596704Z test_scatter_checks (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9596963Z test_scatter_stress (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9597227Z test_scatter_stress_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9597487Z test_send_recv_all_to_all (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9597764Z test_sparse_allreduce_basics (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9598054Z test_sparse_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9598338Z test_sparse_allreduce_checks (__main__.ProcessGroupGlooTest) 2022-11-23T01:41:59.9598997Z , <__main__.ReducerTest testMethod=test_forward_backward_optimizer>, <__main__.ReducerTest testMethod=test_forward_backward_unused_parameters>, <__main__.ReducerTest testMethod=test_multi_dtype_multi_bucket>, <__main__.ReducerTest testMethod=test_multi_dtype_single_bucket>, <__main__.ReducerTest testMethod=test_single_dtype_single_bucket>]> 2022-11-23T01:41:59.9599631Z test_forward_backward (__main__.ReducerTest) 2022-11-23T01:41:59.9599880Z test_forward_backward_optimizer (__main__.ReducerTest) 2022-11-23T01:41:59.9600150Z test_forward_backward_unused_parameters (__main__.ReducerTest) 2022-11-23T01:41:59.9600401Z test_multi_dtype_multi_bucket (__main__.ReducerTest) 2022-11-23T01:41:59.9600833Z test_multi_dtype_single_bucket (__main__.ReducerTest) 2022-11-23T01:41:59.9601092Z test_single_dtype_single_bucket (__main__.ReducerTest) 2022-11-23T01:41:59.9601390Z ]> 2022-11-23T01:41:59.9601693Z test_logging_init (__main__.RendezvousEnvTest) 2022-11-23T01:41:59.9601931Z 2022-11-23T01:41:59.9602226Z ]> 2022-11-23T01:41:59.9602542Z test_default_store_timeout_gloo (__main__.TimeoutTest) 2022-11-23T01:41:59.9603069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9603403Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9603824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9604172Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9604517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyzphaclb 2022-11-23T01:41:59.9604909Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyzphaclb/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9605364Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:41:59.9605560Z 2022-11-23T01:41:59.9605634Z Running tests... 2022-11-23T01:41:59.9605939Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9606311Z test_broadcast_coalesced_gloo_cpu (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2481 2022-11-23T01:41:59.9606691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2482 2022-11-23T01:41:59.9607143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9607475Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9607889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9608237Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9608577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8rcxb1d5 2022-11-23T01:41:59.9609036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8rcxb1d5/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9609420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:41:59.9609883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9610212Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9610625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9610969Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9611315Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjukjamt9 2022-11-23T01:41:59.9611744Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjukjamt9/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9612121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:41:59.9612378Z ok (1.877s) 2022-11-23T01:41:59.9612481Z 2022-11-23T01:41:59.9612679Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9612907Z Ran 1 test in 1.877s 2022-11-23T01:41:59.9613018Z 2022-11-23T01:41:59.9613083Z OK 2022-11-23T01:41:59.9613173Z 2022-11-23T01:41:59.9613257Z Generating XML reports... 2022-11-23T01:41:59.9613649Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123013456.xml 2022-11-23T01:41:59.9614144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9614472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9614899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9615234Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9615577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplc0yb1v4 2022-11-23T01:41:59.9615979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplc0yb1v4/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9616412Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:41:59.9616604Z 2022-11-23T01:41:59.9616675Z Running tests... 2022-11-23T01:41:59.9616973Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9617352Z test_broadcast_coalesced_gloo_cuda (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2520 2022-11-23T01:41:59.9617730Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2521 2022-11-23T01:41:59.9618176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9618505Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9618935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9619270Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9619617Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphussuvz2 2022-11-23T01:41:59.9620018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphussuvz2/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9620380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:41:59.9620840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9621169Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9621597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9621964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9622309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpno7l2j8m 2022-11-23T01:41:59.9622714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpno7l2j8m/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9623075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:41:59.9623365Z skip: Need at least 2 CUDA devices (1.878s) 2022-11-23T01:41:59.9623504Z 2022-11-23T01:41:59.9623703Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9623992Z Ran 1 test in 1.878s 2022-11-23T01:41:59.9624091Z 2022-11-23T01:41:59.9624163Z OK (skipped=1) 2022-11-23T01:41:59.9624269Z 2022-11-23T01:41:59.9624355Z Generating XML reports... 2022-11-23T01:41:59.9624765Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123013459.xml 2022-11-23T01:41:59.9625252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9625580Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9626007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9626354Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9626685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjc921hhe 2022-11-23T01:41:59.9627087Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjc921hhe/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9627532Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:41:59.9627723Z 2022-11-23T01:41:59.9627796Z Running tests... 2022-11-23T01:41:59.9628083Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9628457Z test_gloo_barrier_device_ids (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2553 2022-11-23T01:41:59.9628835Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2554 2022-11-23T01:41:59.9629273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9629604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9630034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9630387Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9630722Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplbnw6p39 2022-11-23T01:41:59.9631123Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplbnw6p39/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9631500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:41:59.9631946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9632274Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9632799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9633145Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9633476Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptk2nfb6k 2022-11-23T01:41:59.9633879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptk2nfb6k/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9634255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:41:59.9634656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:41:59.9635013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:41:59.9635516Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:41:59.9636039Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:41:59.9636321Z ok (1.875s) 2022-11-23T01:41:59.9636424Z 2022-11-23T01:41:59.9636620Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9636890Z Ran 1 test in 1.875s 2022-11-23T01:41:59.9637003Z 2022-11-23T01:41:59.9637065Z OK 2022-11-23T01:41:59.9637142Z 2022-11-23T01:41:59.9637227Z Generating XML reports... 2022-11-23T01:41:59.9637634Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123013503.xml 2022-11-23T01:41:59.9638135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9638459Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9638888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9639234Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9661163Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnp7pc_ur 2022-11-23T01:41:59.9661600Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnp7pc_ur/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9662118Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:41:59.9662316Z 2022-11-23T01:41:59.9662391Z Running tests... 2022-11-23T01:41:59.9662684Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9663064Z test_gloo_rank_membership (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2592 2022-11-23T01:41:59.9663439Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2593 2022-11-23T01:41:59.9663882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9664218Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9664649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9665001Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9665335Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpswf_johw 2022-11-23T01:41:59.9665737Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpswf_johw/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9666118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:41:59.9666568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9666899Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9667325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9667670Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9668004Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk9m8uoad 2022-11-23T01:41:59.9668404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk9m8uoad/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9668779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:41:59.9669069Z skip: Need at least 2 CUDA devices (1.876s) 2022-11-23T01:41:59.9669312Z 2022-11-23T01:41:59.9669518Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9669759Z Ran 1 test in 1.877s 2022-11-23T01:41:59.9669873Z 2022-11-23T01:41:59.9669946Z OK (skipped=1) 2022-11-23T01:41:59.9670054Z 2022-11-23T01:41:59.9670125Z Generating XML reports... 2022-11-23T01:41:59.9670533Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123013506.xml 2022-11-23T01:41:59.9671029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9671359Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9671823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9672167Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9672511Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpitx78ax0 2022-11-23T01:41:59.9672978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpitx78ax0/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9673428Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:41:59.9673620Z 2022-11-23T01:41:59.9673693Z Running tests... 2022-11-23T01:41:59.9673996Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9674353Z test_gloo_warn_not_in_group (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2625 2022-11-23T01:41:59.9674727Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2626 2022-11-23T01:41:59.9675185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9675504Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9675932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9676277Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9676623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpni38d3hk 2022-11-23T01:41:59.9677008Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpni38d3hk/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9677497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9677825Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9678253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9678581Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9678925Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5t1a5iqw 2022-11-23T01:41:59.9679326Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5t1a5iqw/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9679689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:41:59.9680031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:41:59.9680321Z skip: Need at least 2 CUDA devices (1.874s) 2022-11-23T01:41:59.9680456Z 2022-11-23T01:41:59.9680790Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9681023Z Ran 1 test in 1.874s 2022-11-23T01:41:59.9681139Z 2022-11-23T01:41:59.9681212Z OK (skipped=1) 2022-11-23T01:41:59.9681320Z 2022-11-23T01:41:59.9681405Z Generating XML reports... 2022-11-23T01:41:59.9681800Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123013510.xml 2022-11-23T01:41:59.9682355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9682688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9683119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9683454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9683799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjbswev80 2022-11-23T01:41:59.9684201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjbswev80/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9684677Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:41:59.9684872Z 2022-11-23T01:41:59.9684944Z Running tests... 2022-11-23T01:41:59.9685245Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9685634Z test_sequence_num_incremented_gloo_default (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2658 2022-11-23T01:41:59.9686011Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2659 2022-11-23T01:41:59.9686462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9686789Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9690881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9691227Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9691586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpusety3fp 2022-11-23T01:41:59.9691992Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpusety3fp/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9692362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:41:59.9692830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9693161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9693590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9693926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9694275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprk9m6r3_ 2022-11-23T01:41:59.9694681Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprk9m6r3_/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9695060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:41:59.9695337Z skip: Need at least 2 CUDA devices (1.881s) 2022-11-23T01:41:59.9695476Z 2022-11-23T01:41:59.9695681Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9695924Z Ran 1 test in 1.882s 2022-11-23T01:41:59.9696035Z 2022-11-23T01:41:59.9696094Z OK (skipped=1) 2022-11-23T01:41:59.9696202Z 2022-11-23T01:41:59.9696288Z Generating XML reports... 2022-11-23T01:41:59.9696694Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123013513.xml 2022-11-23T01:41:59.9697187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9697505Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9697937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9698283Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9698678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpca_zrm0o 2022-11-23T01:41:59.9699083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpca_zrm0o/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9699528Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:41:59.9699720Z 2022-11-23T01:41:59.9699793Z Running tests... 2022-11-23T01:41:59.9700080Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9700472Z test_sequence_num_incremented_gloo_subgroup (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2691 2022-11-23T01:41:59.9700868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2692 2022-11-23T01:41:59.9701347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9701683Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9702114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9702460Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9702790Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzje5j0ch 2022-11-23T01:41:59.9703191Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzje5j0ch/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9703572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:41:59.9704033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9704354Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9704782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9705130Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9705463Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfouc7joq 2022-11-23T01:41:59.9705863Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfouc7joq/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9706241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:41:59.9706531Z skip: Need at least 4 CUDA devices (1.876s) 2022-11-23T01:41:59.9706655Z 2022-11-23T01:41:59.9706854Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9707096Z Ran 1 test in 1.876s 2022-11-23T01:41:59.9707212Z 2022-11-23T01:41:59.9707285Z OK (skipped=1) 2022-11-23T01:41:59.9707393Z 2022-11-23T01:41:59.9707464Z Generating XML reports... 2022-11-23T01:41:59.9707868Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123013517.xml 2022-11-23T01:41:59.9708366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9708699Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9709115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9709460Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9709807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkikoasvp 2022-11-23T01:41:59.9710198Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkikoasvp/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9710646Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:41:59.9710836Z 2022-11-23T01:41:59.9710910Z Running tests... 2022-11-23T01:41:59.9711211Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9711615Z test_sequence_num_set_default_pg_gloo (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2724 2022-11-23T01:41:59.9712003Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2725 2022-11-23T01:41:59.9712454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9712869Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9713288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9713637Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9714026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3l7249fg 2022-11-23T01:41:59.9714413Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3l7249fg/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9714795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:41:59.9715256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9715582Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9715995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9716340Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9716687Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqkpe_k02 2022-11-23T01:41:59.9717077Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqkpe_k02/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9717459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:41:59.9717749Z skip: Need at least 2 CUDA devices (1.876s) 2022-11-23T01:41:59.9717885Z 2022-11-23T01:41:59.9718086Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9718314Z Ran 1 test in 1.877s 2022-11-23T01:41:59.9718426Z 2022-11-23T01:41:59.9718499Z OK (skipped=1) 2022-11-23T01:41:59.9718608Z 2022-11-23T01:41:59.9718692Z Generating XML reports... 2022-11-23T01:41:59.9719082Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123013520.xml 2022-11-23T01:41:59.9719577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9719908Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9720337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9720847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9721202Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy87cjemv 2022-11-23T01:41:59.9721608Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy87cjemv/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9722060Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:41:59.9722237Z 2022-11-23T01:41:59.9722310Z Running tests... 2022-11-23T01:41:59.9722614Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9722996Z test_sequence_num_set_gloo_new_group (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2757 2022-11-23T01:41:59.9723365Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2758 2022-11-23T01:41:59.9723821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9724153Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9724661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9724998Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9725345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw26cvuqg 2022-11-23T01:41:59.9725747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw26cvuqg/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9726108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:41:59.9726571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9726952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9727379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9727712Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9728058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnegywwzh 2022-11-23T01:41:59.9728465Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnegywwzh/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9728843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:41:59.9729120Z skip: Need at least 2 CUDA devices (1.876s) 2022-11-23T01:41:59.9729257Z 2022-11-23T01:41:59.9729456Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9729696Z Ran 1 test in 1.876s 2022-11-23T01:41:59.9729811Z 2022-11-23T01:41:59.9729870Z OK (skipped=1) 2022-11-23T01:41:59.9729977Z 2022-11-23T01:41:59.9730061Z Generating XML reports... 2022-11-23T01:41:59.9730470Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123013524.xml 2022-11-23T01:41:59.9730966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9731284Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9731709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9732056Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9732387Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzajp6746 2022-11-23T01:41:59.9732788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzajp6746/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9733233Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:41:59.9733424Z 2022-11-23T01:41:59.9733497Z Running tests... 2022-11-23T01:41:59.9733784Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9734156Z test_tensor_dtype_complex (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2790 2022-11-23T01:41:59.9734531Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2791 2022-11-23T01:41:59.9734969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9735301Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9735726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9736069Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9736400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4is88eov 2022-11-23T01:41:59.9736798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4is88eov/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9737311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9737642Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9738057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9738405Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9738750Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqy0y6g2l 2022-11-23T01:41:59.9739135Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqy0y6g2l/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9739544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:41:59.9739892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:41:59.9740180Z skip: Need at least 2 CUDA devices (1.874s) 2022-11-23T01:41:59.9740303Z 2022-11-23T01:41:59.9740506Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9740744Z Ran 1 test in 1.874s 2022-11-23T01:41:59.9740856Z 2022-11-23T01:41:59.9740928Z OK (skipped=1) 2022-11-23T01:41:59.9741035Z 2022-11-23T01:41:59.9741105Z Generating XML reports... 2022-11-23T01:41:59.9741506Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123013527.xml 2022-11-23T01:41:59.9742003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9742333Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9742749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9743094Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9743443Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp34zy0y24 2022-11-23T01:41:59.9743830Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp34zy0y24/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9744270Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:41:59.9744464Z 2022-11-23T01:41:59.9744537Z Running tests... 2022-11-23T01:41:59.9744838Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9745194Z test_tensor_dtype_mismatch (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2823 2022-11-23T01:41:59.9745569Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2824 2022-11-23T01:41:59.9746019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9746336Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9746765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9747113Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9747460Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfqai86zh 2022-11-23T01:41:59.9747848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfqai86zh/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9748332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9748659Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9749088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9749420Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9749795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9zpb8opy 2022-11-23T01:41:59.9750203Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9zpb8opy/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9750567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:41:59.9750916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:41:59.9751203Z skip: Need at least 2 CUDA devices (1.877s) 2022-11-23T01:41:59.9751339Z 2022-11-23T01:41:59.9751536Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9751761Z Ran 1 test in 1.877s 2022-11-23T01:41:59.9751874Z 2022-11-23T01:41:59.9751980Z OK (skipped=1) 2022-11-23T01:41:59.9752091Z 2022-11-23T01:41:59.9752175Z Generating XML reports... 2022-11-23T01:41:59.9752567Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123013531.xml 2022-11-23T01:41:59.9753148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9753479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9753907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9754238Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9754582Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_nhy9vag 2022-11-23T01:41:59.9754982Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_nhy9vag/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9755411Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:41:59.9755601Z 2022-11-23T01:41:59.9755675Z Running tests... 2022-11-23T01:41:59.9755979Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9756361Z test_allgather_work_wait_cpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2856 2022-11-23T01:41:59.9756730Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2857 2022-11-23T01:41:59.9757183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9757514Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9757941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9758273Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9758620Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1fjgq_jz 2022-11-23T01:41:59.9759097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9759414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9759840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9760184Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9760529Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy6a8028k 2022-11-23T01:41:59.9761078Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1fjgq_jz/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9761484Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy6a8028k/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9761862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:41:59.9762195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:41:59.9762560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:41:59.9762990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:41:59.9763492Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:41:59.9763999Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:41:59.9764708Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T01:41:59.9765305Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T01:41:59.9765959Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T01:41:59.9766498Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T01:41:59.9766726Z ok (1.878s) 2022-11-23T01:41:59.9766830Z 2022-11-23T01:41:59.9767028Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9767270Z Ran 1 test in 1.878s 2022-11-23T01:41:59.9767382Z 2022-11-23T01:41:59.9767430Z OK 2022-11-23T01:41:59.9767522Z 2022-11-23T01:41:59.9767608Z Generating XML reports... 2022-11-23T01:41:59.9768032Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123013534.xml 2022-11-23T01:41:59.9768543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9768861Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9769298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9769646Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9769979Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv7h8golc 2022-11-23T01:41:59.9770383Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv7h8golc/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9770829Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:41:59.9771020Z 2022-11-23T01:41:59.9771093Z Running tests... 2022-11-23T01:41:59.9771383Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9771767Z test_allgather_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2895 2022-11-23T01:41:59.9772151Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2896 2022-11-23T01:41:59.9772592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9772926Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9773354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9773701Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9774033Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphd2pmeu8 2022-11-23T01:41:59.9774436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphd2pmeu8/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9774922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9775255Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9775704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9776056Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9776397Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6gb29c4f 2022-11-23T01:41:59.9776783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6gb29c4f/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9777164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:41:59.9777512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:41:59.9777831Z skip: Need at least 2 CUDA devices (1.874s) 2022-11-23T01:41:59.9777953Z 2022-11-23T01:41:59.9778150Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9778391Z Ran 1 test in 1.874s 2022-11-23T01:41:59.9778503Z 2022-11-23T01:41:59.9778575Z OK (skipped=1) 2022-11-23T01:41:59.9778683Z 2022-11-23T01:41:59.9778757Z Generating XML reports... 2022-11-23T01:41:59.9779172Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123013538.xml 2022-11-23T01:41:59.9779681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9780015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9780429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9780776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9781123Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmqqfmrc0 2022-11-23T01:41:59.9781512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmqqfmrc0/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9781957Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:41:59.9782149Z 2022-11-23T01:41:59.9782224Z Running tests... 2022-11-23T01:41:59.9782525Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9782889Z test_allreduce_work_wait_cpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2928 2022-11-23T01:41:59.9783272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2929 2022-11-23T01:41:59.9783724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9784053Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9784472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9784820Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9785166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4i_7j9v2 2022-11-23T01:41:59.9785554Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4i_7j9v2/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9786037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9786368Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9786796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9787128Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9787472Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaws7qq8w 2022-11-23T01:41:59.9787872Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaws7qq8w/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9788267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:41:59.9788618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:41:59.9788975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:41:59.9789350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:41:59.9789831Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:41:59.9790348Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:41:59.9791084Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T01:41:59.9791635Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T01:41:59.9792272Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T01:41:59.9792882Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T01:41:59.9793534Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T01:41:59.9794072Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T01:41:59.9794715Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T01:41:59.9795238Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T01:41:59.9795475Z ok (1.974s) 2022-11-23T01:41:59.9795580Z 2022-11-23T01:41:59.9795780Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9796008Z Ran 1 test in 1.974s 2022-11-23T01:41:59.9796121Z 2022-11-23T01:41:59.9796181Z OK 2022-11-23T01:41:59.9796272Z 2022-11-23T01:41:59.9796356Z Generating XML reports... 2022-11-23T01:41:59.9796779Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123013541.xml 2022-11-23T01:41:59.9797272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9797602Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9798036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9798370Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9798718Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkhn31757 2022-11-23T01:41:59.9799119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkhn31757/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9799558Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:41:59.9799748Z 2022-11-23T01:41:59.9799807Z Running tests... 2022-11-23T01:41:59.9800113Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9800493Z test_allreduce_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2967 2022-11-23T01:41:59.9801021Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2968 2022-11-23T01:41:59.9801526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9801863Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9802300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9802634Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9802980Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0afy_bq4 2022-11-23T01:41:59.9803380Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0afy_bq4/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9803906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9804225Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9804655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9805004Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9805334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpql7svt4q 2022-11-23T01:41:59.9805735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpql7svt4q/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9806114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:41:59.9806462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:41:59.9806740Z skip: Need at least 2 CUDA devices (1.782s) 2022-11-23T01:41:59.9806877Z 2022-11-23T01:41:59.9807075Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9807316Z Ran 1 test in 1.782s 2022-11-23T01:41:59.9807428Z 2022-11-23T01:41:59.9807486Z OK (skipped=1) 2022-11-23T01:41:59.9807597Z 2022-11-23T01:41:59.9807683Z Generating XML reports... 2022-11-23T01:41:59.9808102Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123013545.xml 2022-11-23T01:41:59.9808608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9808922Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9809351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9809700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9810047Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3_uhb9sb 2022-11-23T01:41:59.9810436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3_uhb9sb/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9810882Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:41:59.9811075Z 2022-11-23T01:41:59.9811147Z Running tests... 2022-11-23T01:41:59.9811437Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9811814Z test_broadcast_work_wait_cpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3000 2022-11-23T01:41:59.9812197Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3001 2022-11-23T01:41:59.9812647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9812966Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9813394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9813741Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9814127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjf384qg4 2022-11-23T01:41:59.9814612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9814939Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9815368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9815699Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9816045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4gkiucag 2022-11-23T01:41:59.9816481Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjf384qg4/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9816890Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4gkiucag/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9817256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:41:59.9817605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:41:59.9817974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:41:59.9818334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:41:59.9818824Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:41:59.9819347Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:41:59.9820056Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T01:41:59.9820588Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T01:41:59.9821235Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T01:41:59.9821767Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T01:41:59.9822007Z ok (1.873s) 2022-11-23T01:41:59.9822111Z 2022-11-23T01:41:59.9822297Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9822540Z Ran 1 test in 1.873s 2022-11-23T01:41:59.9822655Z 2022-11-23T01:41:59.9822717Z OK 2022-11-23T01:41:59.9822809Z 2022-11-23T01:41:59.9822893Z Generating XML reports... 2022-11-23T01:41:59.9823296Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123013548.xml 2022-11-23T01:41:59.9823809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9824141Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9824554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9824909Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9825266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpznomh_rw 2022-11-23T01:41:59.9825668Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpznomh_rw/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9826106Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:41:59.9826298Z 2022-11-23T01:41:59.9826371Z Running tests... 2022-11-23T01:41:59.9826672Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9827070Z test_broadcast_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3039 2022-11-23T01:41:59.9827455Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3040 2022-11-23T01:41:59.9827907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9828240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9828655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9829036Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9829385Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgvp9zxx6 2022-11-23T01:41:59.9829790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgvp9zxx6/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9830158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:41:59.9830624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9830954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9831370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9831716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9832060Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzgjoph62 2022-11-23T01:41:59.9832468Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzgjoph62/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9832916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:41:59.9833208Z skip: Need at least 2 CUDA devices (1.879s) 2022-11-23T01:41:59.9833347Z 2022-11-23T01:41:59.9833550Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9833776Z Ran 1 test in 1.879s 2022-11-23T01:41:59.9833890Z 2022-11-23T01:41:59.9833966Z OK (skipped=1) 2022-11-23T01:41:59.9834074Z 2022-11-23T01:41:59.9834159Z Generating XML reports... 2022-11-23T01:41:59.9834574Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123013552.xml 2022-11-23T01:41:59.9835065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9835398Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9835830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9836176Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9836511Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk8ozstnw 2022-11-23T01:41:59.9836919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk8ozstnw/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9837364Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:41:59.9837554Z 2022-11-23T01:41:59.9837615Z Running tests... 2022-11-23T01:41:59.9837917Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9838302Z test_consecutive_comm_work_wait_cpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3072 2022-11-23T01:41:59.9838694Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3073 2022-11-23T01:41:59.9839135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9839467Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9839943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9840278Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9840746Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2u8sg4jz 2022-11-23T01:41:59.9841156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2u8sg4jz/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9841528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:41:59.9841979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9842364Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9842795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9843142Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9843473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzlzrsxak 2022-11-23T01:41:59.9843875Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzlzrsxak/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9844255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:41:59.9844605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:41:59.9844974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:41:59.9845471Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:41:59.9845995Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:41:59.9846688Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T01:41:59.9847235Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T01:41:59.9847883Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T01:41:59.9848422Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T01:41:59.9849055Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T01:41:59.9849595Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T01:41:59.9850237Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T01:41:59.9850770Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T01:41:59.9851415Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T01:41:59.9851937Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T01:41:59.9852627Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T01:41:59.9853165Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T01:41:59.9853809Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T01:41:59.9854344Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T01:41:59.9854999Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T01:41:59.9855532Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T01:41:59.9855773Z ok (1.876s) 2022-11-23T01:41:59.9855876Z 2022-11-23T01:41:59.9856063Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9856303Z Ran 1 test in 1.876s 2022-11-23T01:41:59.9856418Z 2022-11-23T01:41:59.9856480Z OK 2022-11-23T01:41:59.9856571Z 2022-11-23T01:41:59.9856655Z Generating XML reports... 2022-11-23T01:41:59.9857057Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123013555.xml 2022-11-23T01:41:59.9857568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9857903Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9858321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9858674Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9859021Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm4rwf8cj 2022-11-23T01:41:59.9859425Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm4rwf8cj/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9859861Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:41:59.9860053Z 2022-11-23T01:41:59.9860127Z Running tests... 2022-11-23T01:41:59.9860428Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9860814Z test_consecutive_comm_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3111 2022-11-23T01:41:59.9861191Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3112 2022-11-23T01:41:59.9861646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9861983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9862399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9862749Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9863099Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnn98c1_9 2022-11-23T01:41:59.9863501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnn98c1_9/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9863969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9864300Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9864728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9865096Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9865445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvf6_8_km 2022-11-23T01:41:59.9865847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvf6_8_km/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9866228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:41:59.9866564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:41:59.9866850Z skip: Need at least 2 CUDA devices (1.877s) 2022-11-23T01:41:59.9866988Z 2022-11-23T01:41:59.9867227Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9867468Z Ran 1 test in 1.877s 2022-11-23T01:41:59.9867568Z 2022-11-23T01:41:59.9867640Z OK (skipped=1) 2022-11-23T01:41:59.9867747Z 2022-11-23T01:41:59.9867832Z Generating XML reports... 2022-11-23T01:41:59.9868251Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123013559.xml 2022-11-23T01:41:59.9868743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9869080Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9869507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9869852Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9870183Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwt92n4h3 2022-11-23T01:41:59.9870589Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwt92n4h3/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9871033Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:41:59.9871224Z 2022-11-23T01:41:59.9871283Z Running tests... 2022-11-23T01:41:59.9871588Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9871974Z test_nested_comm_tensor_wrapping (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3144 2022-11-23T01:41:59.9872361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3145 2022-11-23T01:41:59.9872877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9873209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9873640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9873978Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9874327Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfrv97o5n 2022-11-23T01:41:59.9874730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfrv97o5n/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9875107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:41:59.9875555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9875887Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9876319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9876673Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9877003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7_j39ezl 2022-11-23T01:41:59.9877402Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7_j39ezl/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9877833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:41:59.9878184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:41:59.9878551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:41:59.9879044Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:41:59.9879568Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:41:59.9880271Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T01:41:59.9880986Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T01:41:59.9881643Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T01:41:59.9882183Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T01:41:59.9882826Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T01:41:59.9883352Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T01:41:59.9884001Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T01:41:59.9884536Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T01:41:59.9884774Z ok (1.874s) 2022-11-23T01:41:59.9884864Z 2022-11-23T01:41:59.9885064Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9885308Z Ran 1 test in 1.874s 2022-11-23T01:41:59.9885421Z 2022-11-23T01:41:59.9885482Z OK 2022-11-23T01:41:59.9885574Z 2022-11-23T01:41:59.9885644Z Generating XML reports... 2022-11-23T01:41:59.9886064Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123013602.xml 2022-11-23T01:41:59.9886570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9886906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9887325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9887674Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9888024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph1vrb9qx 2022-11-23T01:41:59.9888415Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph1vrb9qx/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9888865Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:41:59.9889055Z 2022-11-23T01:41:59.9889128Z Running tests... 2022-11-23T01:41:59.9889429Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9889795Z test_scatter_work_wait_cpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3183 2022-11-23T01:41:59.9890173Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3184 2022-11-23T01:41:59.9890688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9891008Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9891443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9891794Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9892141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq9snlpvj 2022-11-23T01:41:59.9892529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq9snlpvj/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9893020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9893395Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9893827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9894161Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9894506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphxmvd42l 2022-11-23T01:41:59.9894906Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphxmvd42l/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9895272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:41:59.9895621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:41:59.9895978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:41:59.9896355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:41:59.9896836Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:41:59.9897358Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:41:59.9898067Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T01:41:59.9898606Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T01:41:59.9899241Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T01:41:59.9899780Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T01:41:59.9900018Z ok (1.872s) 2022-11-23T01:41:59.9900120Z 2022-11-23T01:41:59.9900320Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9900549Z Ran 1 test in 1.872s 2022-11-23T01:41:59.9900661Z 2022-11-23T01:41:59.9900722Z OK 2022-11-23T01:41:59.9900813Z 2022-11-23T01:41:59.9900897Z Generating XML reports... 2022-11-23T01:41:59.9901306Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123013606.xml 2022-11-23T01:41:59.9901809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9902141Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9902576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9902913Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9903258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp00_xonw5 2022-11-23T01:41:59.9903717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp00_xonw5/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9904167Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:41:59.9904346Z 2022-11-23T01:41:59.9904420Z Running tests... 2022-11-23T01:41:59.9904724Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9905103Z test_scatter_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3222 2022-11-23T01:41:59.9905472Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3223 2022-11-23T01:41:59.9905958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9906291Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9906727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9907065Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9907414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwdma9o21 2022-11-23T01:41:59.9907818Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwdma9o21/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9908181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:41:59.9908641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9908975Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9909407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9909742Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9910092Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz7nrr0hr 2022-11-23T01:41:59.9910494Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz7nrr0hr/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9910876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:41:59.9911152Z skip: Need at least 2 CUDA devices (1.875s) 2022-11-23T01:41:59.9911289Z 2022-11-23T01:41:59.9911490Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9911728Z Ran 1 test in 1.876s 2022-11-23T01:41:59.9911840Z 2022-11-23T01:41:59.9911899Z OK (skipped=1) 2022-11-23T01:41:59.9912008Z 2022-11-23T01:41:59.9912093Z Generating XML reports... 2022-11-23T01:41:59.9912510Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123013609.xml 2022-11-23T01:41:59.9913109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9913428Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9913859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9914206Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9914542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo9e2nkx3 2022-11-23T01:41:59.9914942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo9e2nkx3/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9915388Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:41:59.9915582Z 2022-11-23T01:41:59.9915658Z Running tests... 2022-11-23T01:41:59.9915944Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9916276Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9916846Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3255 2022-11-23T01:41:59.9917241Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3256 2022-11-23T01:41:59.9917690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9918024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9918450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9918819Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9919164Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt_yh0ehy 2022-11-23T01:41:59.9919649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9919979Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9920396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9920887Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9921235Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4sio9nqt 2022-11-23T01:41:59.9921623Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt_yh0ehy/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9922028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4sio9nqt/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9922410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:41:59.9922762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:41:59.9923038Z skip: Need at least 2 CUDA devices (1.872s) 2022-11-23T01:41:59.9923175Z 2022-11-23T01:41:59.9923377Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9923621Z Ran 1 test in 1.872s 2022-11-23T01:41:59.9923732Z 2022-11-23T01:41:59.9923790Z OK (skipped=1) 2022-11-23T01:41:59.9923897Z 2022-11-23T01:41:59.9923982Z Generating XML reports... 2022-11-23T01:41:59.9924445Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013613.xml 2022-11-23T01:41:59.9924994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9925312Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9925739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9926089Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9926424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9y5s_utq 2022-11-23T01:41:59.9926826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9y5s_utq/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9927270Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:41:59.9927461Z 2022-11-23T01:41:59.9927535Z Running tests... 2022-11-23T01:41:59.9927822Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9928162Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9928599Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3288 2022-11-23T01:41:59.9929000Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3289 2022-11-23T01:41:59.9929506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9929841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9930270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9930602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9930950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6xjnfi24 2022-11-23T01:41:59.9931352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6xjnfi24/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9931774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:41:59.9932217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9932547Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9932981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9933314Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9933659Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgkkzn0p4 2022-11-23T01:41:59.9934061Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgkkzn0p4/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9934436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:41:59.9934715Z skip: Need at least 2 CUDA devices (1.877s) 2022-11-23T01:41:59.9934854Z 2022-11-23T01:41:59.9935054Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9935294Z Ran 1 test in 1.878s 2022-11-23T01:41:59.9935406Z 2022-11-23T01:41:59.9935464Z OK (skipped=1) 2022-11-23T01:41:59.9935571Z 2022-11-23T01:41:59.9935658Z Generating XML reports... 2022-11-23T01:41:59.9936125Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013616.xml 2022-11-23T01:41:59.9936671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9936990Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9937418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9937763Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9938109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpetdzq6ex 2022-11-23T01:41:59.9938499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpetdzq6ex/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9938946Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:41:59.9939140Z 2022-11-23T01:41:59.9939213Z Running tests... 2022-11-23T01:41:59.9939502Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9939839Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9940260Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3321 2022-11-23T01:41:59.9940652Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3322 2022-11-23T01:41:59.9941089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9941423Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9941850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9942215Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9942561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkm95_2yl 2022-11-23T01:41:59.9942964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkm95_2yl/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9943452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9943771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9944200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9944581Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9944929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps3yw26ve 2022-11-23T01:41:59.9945314Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps3yw26ve/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9945697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:41:59.9946044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:41:59.9946321Z skip: Need at least 2 CUDA devices (1.878s) 2022-11-23T01:41:59.9946457Z 2022-11-23T01:41:59.9946653Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9946896Z Ran 1 test in 1.878s 2022-11-23T01:41:59.9947008Z 2022-11-23T01:41:59.9947079Z OK (skipped=1) 2022-11-23T01:41:59.9947173Z 2022-11-23T01:41:59.9947259Z Generating XML reports... 2022-11-23T01:41:59.9947733Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013620.xml 2022-11-23T01:41:59.9948407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9948887Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9949579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9949927Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9950277Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdzlmu8zx 2022-11-23T01:41:59.9950667Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdzlmu8zx/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9951115Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:41:59.9951310Z 2022-11-23T01:41:59.9951384Z Running tests... 2022-11-23T01:41:59.9951671Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9952010Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9952432Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3354 2022-11-23T01:41:59.9952889Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3355 2022-11-23T01:41:59.9953330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9953663Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9954100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9954449Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9954785Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplwfdkp_u 2022-11-23T01:41:59.9955199Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplwfdkp_u/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9955743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9956060Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9956489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9956835Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9957179Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_b99hsoh 2022-11-23T01:41:59.9957567Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_b99hsoh/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9957977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:41:59.9958326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:41:59.9958602Z skip: Need at least 2 CUDA devices (1.881s) 2022-11-23T01:41:59.9958739Z 2022-11-23T01:41:59.9958940Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9959181Z Ran 1 test in 1.881s 2022-11-23T01:41:59.9959294Z 2022-11-23T01:41:59.9959366Z OK (skipped=1) 2022-11-23T01:41:59.9959472Z 2022-11-23T01:41:59.9959543Z Generating XML reports... 2022-11-23T01:41:59.9960007Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013623.xml 2022-11-23T01:41:59.9960552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9961025Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9961450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9961798Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9962146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgf6q98ax 2022-11-23T01:41:59.9962538Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgf6q98ax/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9962985Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:41:59.9962990Z 2022-11-23T01:41:59.9963064Z Running tests... 2022-11-23T01:41:59.9963263Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9963458Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9963787Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3387 2022-11-23T01:41:59.9963947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3388 2022-11-23T01:41:59.9964206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9964339Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9964623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9964761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9964946Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp62yf39ir 2022-11-23T01:41:59.9965142Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp62yf39ir/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9965418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9965544Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9965824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9966016Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9966206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpetkerdgv 2022-11-23T01:41:59.9966405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpetkerdgv/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9966569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:41:59.9966733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:41:59.9966837Z skip: Need at least 2 CUDA devices (1.885s) 2022-11-23T01:41:59.9966842Z 2022-11-23T01:41:59.9967040Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9967154Z Ran 1 test in 1.885s 2022-11-23T01:41:59.9967159Z 2022-11-23T01:41:59.9967217Z OK (skipped=1) 2022-11-23T01:41:59.9967222Z 2022-11-23T01:41:59.9967307Z Generating XML reports... 2022-11-23T01:41:59.9967660Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013627.xml 2022-11-23T01:41:59.9967934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9968059Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9968341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9968477Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9968661Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4s_6mis_ 2022-11-23T01:41:59.9968859Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4s_6mis_/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9969076Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:41:59.9969082Z 2022-11-23T01:41:59.9969156Z Running tests... 2022-11-23T01:41:59.9969356Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9969548Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9969877Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3420 2022-11-23T01:41:59.9970033Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3421 2022-11-23T01:41:59.9970307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9970431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9970701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9970838Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9971027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0exmn_yv 2022-11-23T01:41:59.9971223Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0exmn_yv/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9971497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9971622Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9971901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9972037Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9972226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8lgh0xx5 2022-11-23T01:41:59.9972408Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8lgh0xx5/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9972601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:41:59.9972767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:41:59.9972872Z skip: Need at least 2 CUDA devices (1.874s) 2022-11-23T01:41:59.9972878Z 2022-11-23T01:41:59.9973075Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9973152Z Ran 1 test in 1.874s 2022-11-23T01:41:59.9973157Z 2022-11-23T01:41:59.9973229Z OK (skipped=1) 2022-11-23T01:41:59.9973233Z 2022-11-23T01:41:59.9973318Z Generating XML reports... 2022-11-23T01:41:59.9973652Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013630.xml 2022-11-23T01:41:59.9973950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9974073Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9974356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9974493Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9974680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxpngzs35 2022-11-23T01:41:59.9974875Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxpngzs35/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9975106Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:41:59.9975110Z 2022-11-23T01:41:59.9975184Z Running tests... 2022-11-23T01:41:59.9975366Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9975543Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9975893Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3453 2022-11-23T01:41:59.9976049Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3454 2022-11-23T01:41:59.9976323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9976447Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9976729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9976864Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9977050Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6c41c7qz 2022-11-23T01:41:59.9977233Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6c41c7qz/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9977504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9977630Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9977912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9978050Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9978236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu_ejpfnb 2022-11-23T01:41:59.9978430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu_ejpfnb/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9978594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:41:59.9978743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:41:59.9978849Z skip: Need at least 2 CUDA devices (1.878s) 2022-11-23T01:41:59.9978853Z 2022-11-23T01:41:59.9979048Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9979170Z Ran 1 test in 1.879s 2022-11-23T01:41:59.9979176Z 2022-11-23T01:41:59.9979253Z OK (skipped=1) 2022-11-23T01:41:59.9979257Z 2022-11-23T01:41:59.9979341Z Generating XML reports... 2022-11-23T01:41:59.9979690Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013634.xml 2022-11-23T01:41:59.9979964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9980088Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9980352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9980519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9980705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsumk5qe0 2022-11-23T01:41:59.9980906Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsumk5qe0/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9981136Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:41:59.9981141Z 2022-11-23T01:41:59.9981215Z Running tests... 2022-11-23T01:41:59.9981411Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9981587Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9981938Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3486 2022-11-23T01:41:59.9982083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3487 2022-11-23T01:41:59.9982358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9982485Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9982766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9982902Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9983088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxc6e5zfn 2022-11-23T01:41:59.9983284Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxc6e5zfn/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9983449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:41:59.9983708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9983836Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9984115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9984252Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9984438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkbd_w9qy 2022-11-23T01:41:59.9984635Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkbd_w9qy/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9984799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:41:59.9984904Z skip: Need at least 2 CUDA devices (1.874s) 2022-11-23T01:41:59.9984909Z 2022-11-23T01:41:59.9985105Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9985170Z Ran 1 test in 1.874s 2022-11-23T01:41:59.9985175Z 2022-11-23T01:41:59.9985246Z OK (skipped=1) 2022-11-23T01:41:59.9985251Z 2022-11-23T01:41:59.9985335Z Generating XML reports... 2022-11-23T01:41:59.9985682Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013637.xml 2022-11-23T01:41:59.9985985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9986112Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9986396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9986532Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9986704Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4zs4luiz 2022-11-23T01:41:59.9986901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4zs4luiz/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9987162Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:41:59.9987167Z 2022-11-23T01:41:59.9987240Z Running tests... 2022-11-23T01:41:59.9987437Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9987608Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9987844Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3519 2022-11-23T01:41:59.9988002Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3520 2022-11-23T01:41:59.9988275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9988386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9988670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9988804Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9988991Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9mzw_08v 2022-11-23T01:41:59.9989261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9989382Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9989657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9989788Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9989967Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_2jdt_4n 2022-11-23T01:41:59.9990158Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9mzw_08v/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9990334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_2jdt_4n/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9990499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:41:59.9990662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:41:59.9990765Z skip: Need at least 2 CUDA devices (1.873s) 2022-11-23T01:41:59.9990770Z 2022-11-23T01:41:59.9990963Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9991038Z Ran 1 test in 1.873s 2022-11-23T01:41:59.9991043Z 2022-11-23T01:41:59.9991114Z OK (skipped=1) 2022-11-23T01:41:59.9991119Z 2022-11-23T01:41:59.9991201Z Generating XML reports... 2022-11-23T01:41:59.9991547Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013641.xml 2022-11-23T01:41:59.9991810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9991932Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9992244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9992379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9992561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp45uuorif 2022-11-23T01:41:59.9992828Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp45uuorif/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9993066Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:41:59.9993072Z 2022-11-23T01:41:59.9993143Z Running tests... 2022-11-23T01:41:59.9993325Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9993547Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9993784Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3552 2022-11-23T01:41:59.9993940Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3553 2022-11-23T01:41:59.9994215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9994338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9994615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9994750Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9994939Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsp5hvger 2022-11-23T01:41:59.9995125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsp5hvger/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9995287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:41:59.9995558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9995680Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9995959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9996092Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9996275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptjeaugps 2022-11-23T01:41:59.9996471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptjeaugps/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9996635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:41:59.9996728Z skip: Need at least 2 CUDA devices (1.877s) 2022-11-23T01:41:59.9996734Z 2022-11-23T01:41:59.9996928Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9997002Z Ran 1 test in 1.877s 2022-11-23T01:41:59.9997010Z 2022-11-23T01:41:59.9997081Z OK (skipped=1) 2022-11-23T01:41:59.9997086Z 2022-11-23T01:41:59.9997168Z Generating XML reports... 2022-11-23T01:41:59.9997516Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013644.xml 2022-11-23T01:41:59.9997788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:41:59.9997914Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:41:59.9998180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:41:59.9998318Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:41:59.9998499Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_my9x5fi 2022-11-23T01:41:59.9998727Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_my9x5fi/_remote_module_non_scriptable.py 2022-11-23T01:41:59.9998960Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:41:59.9998966Z 2022-11-23T01:41:59.9999038Z Running tests... 2022-11-23T01:41:59.9999230Z ---------------------------------------------------------------------- 2022-11-23T01:41:59.9999414Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T01:41:59.9999649Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3585 2022-11-23T01:41:59.9999791Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3586 2022-11-23T01:42:00.0000093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0000214Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0000491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0000762Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0000952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwm6znfkk 2022-11-23T01:42:00.0001149Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwm6znfkk/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0001423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0001546Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0001818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0001950Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0002135Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplyw8kz_5 2022-11-23T01:42:00.0002330Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplyw8kz_5/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0002491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0002652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0002754Z skip: Need at least 2 CUDA devices (1.877s) 2022-11-23T01:42:00.0002759Z 2022-11-23T01:42:00.0002953Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0003015Z Ran 1 test in 1.877s 2022-11-23T01:42:00.0003035Z 2022-11-23T01:42:00.0003094Z OK (skipped=1) 2022-11-23T01:42:00.0003099Z 2022-11-23T01:42:00.0003180Z Generating XML reports... 2022-11-23T01:42:00.0003529Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013648.xml 2022-11-23T01:42:00.0003801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0003924Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0004203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0004336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0004521Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdiumriac 2022-11-23T01:42:00.0004707Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdiumriac/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0004938Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0004943Z 2022-11-23T01:42:00.0005014Z Running tests... 2022-11-23T01:42:00.0005207Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0005460Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T01:42:00.0005675Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3618 2022-11-23T01:42:00.0005828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3619 2022-11-23T01:42:00.0006099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0006221Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0006488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0006659Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0006846Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9p6ojlnv 2022-11-23T01:42:00.0007044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9p6ojlnv/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0007312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0007434Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0007712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0007845Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0008018Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkvk2vp_t 2022-11-23T01:42:00.0008218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkvk2vp_t/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0008381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0008543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0008646Z skip: Need at least 2 CUDA devices (1.773s) 2022-11-23T01:42:00.0008651Z 2022-11-23T01:42:00.0008847Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0008922Z Ran 1 test in 1.773s 2022-11-23T01:42:00.0008927Z 2022-11-23T01:42:00.0008998Z OK (skipped=1) 2022-11-23T01:42:00.0009002Z 2022-11-23T01:42:00.0009084Z Generating XML reports... 2022-11-23T01:42:00.0009420Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013651.xml 2022-11-23T01:42:00.0009694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0009820Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0010097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0010233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0010417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv0rbxp7r 2022-11-23T01:42:00.0010610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv0rbxp7r/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0010837Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0010842Z 2022-11-23T01:42:00.0010902Z Running tests... 2022-11-23T01:42:00.0011095Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0011280Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T01:42:00.0011495Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3651 2022-11-23T01:42:00.0011649Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3652 2022-11-23T01:42:00.0011956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0012081Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0012363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0012495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0012667Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn361_ces 2022-11-23T01:42:00.0012937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0013089Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0013366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0013502Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0013686Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpprin_q36 2022-11-23T01:42:00.0013877Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn361_ces/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0014066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpprin_q36/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0014218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0014380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0014483Z skip: Need at least 2 CUDA devices (1.874s) 2022-11-23T01:42:00.0014488Z 2022-11-23T01:42:00.0014677Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0014752Z Ran 1 test in 1.874s 2022-11-23T01:42:00.0014757Z 2022-11-23T01:42:00.0014827Z OK (skipped=1) 2022-11-23T01:42:00.0014832Z 2022-11-23T01:42:00.0014915Z Generating XML reports... 2022-11-23T01:42:00.0015260Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013655.xml 2022-11-23T01:42:00.0015530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0015642Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0015917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0016048Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0016235Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx8x43t0d 2022-11-23T01:42:00.0016433Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx8x43t0d/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0016665Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0016670Z 2022-11-23T01:42:00.0016740Z Running tests... 2022-11-23T01:42:00.0016935Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0017097Z test_ddp_comm_hook_future_passing_cpu (__main__.DistributedDataParallelTest) 2022-11-23T01:42:00.0017321Z This unit test verifies whether the Future object is passed properly. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3684 2022-11-23T01:42:00.0017474Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3685 2022-11-23T01:42:00.0017745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0017870Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0018145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0018319Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0018505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprnq2_tsx 2022-11-23T01:42:00.0018699Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprnq2_tsx/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0018958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0019077Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0019357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0019524Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0019707Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxb86mxtc 2022-11-23T01:42:00.0019902Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxb86mxtc/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0020062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0020224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0020289Z ok (1.873s) 2022-11-23T01:42:00.0020294Z 2022-11-23T01:42:00.0020475Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0020552Z Ran 1 test in 1.873s 2022-11-23T01:42:00.0020557Z 2022-11-23T01:42:00.0020617Z OK 2022-11-23T01:42:00.0020621Z 2022-11-23T01:42:00.0020704Z Generating XML reports... 2022-11-23T01:42:00.0021053Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013658.xml 2022-11-23T01:42:00.0021323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0021447Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0021724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0021847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0022030Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpenoeckqw 2022-11-23T01:42:00.0022226Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpenoeckqw/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0022454Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0022461Z 2022-11-23T01:42:00.0022532Z Running tests... 2022-11-23T01:42:00.0022727Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0022890Z test_ddp_comm_hook_future_passing_gpu_gloo (__main__.DistributedDataParallelTest) 2022-11-23T01:42:00.0023144Z This unit test verifies whether the Future object is passed properly using gloo backend. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3723 2022-11-23T01:42:00.0023298Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3724 2022-11-23T01:42:00.0023557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0023678Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0023957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0024092Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0024278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp12tvofgh 2022-11-23T01:42:00.0024477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp12tvofgh/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0024685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0024962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0025080Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0025347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0025482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0025666Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyst45igm 2022-11-23T01:42:00.0025891Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyst45igm/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0026050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0026151Z skip: Need at least 2 CUDA devices (1.883s) 2022-11-23T01:42:00.0026157Z 2022-11-23T01:42:00.0026355Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0026428Z Ran 1 test in 1.883s 2022-11-23T01:42:00.0026433Z 2022-11-23T01:42:00.0026491Z OK (skipped=1) 2022-11-23T01:42:00.0026496Z 2022-11-23T01:42:00.0026577Z Generating XML reports... 2022-11-23T01:42:00.0026925Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013702.xml 2022-11-23T01:42:00.0027197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0027319Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0027596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0027730Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0027913Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz5akfixd 2022-11-23T01:42:00.0028107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz5akfixd/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0028323Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0028328Z 2022-11-23T01:42:00.0028400Z Running tests... 2022-11-23T01:42:00.0028594Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0028751Z test_ddp_comm_hook_register_just_once (__main__.DistributedDataParallelTest) 2022-11-23T01:42:00.0028997Z DDP communication hook can only be registered once. This test validates whether ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3756 2022-11-23T01:42:00.0029152Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3757 2022-11-23T01:42:00.0029427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0029550Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0029817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0029952Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0030135Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcs9o3dj5 2022-11-23T01:42:00.0030328Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcs9o3dj5/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0030489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0030761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0030881Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0031191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0031325Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0031496Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj2pt6xvq 2022-11-23T01:42:00.0031690Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj2pt6xvq/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0031850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0031918Z ok (1.874s) 2022-11-23T01:42:00.0031923Z 2022-11-23T01:42:00.0032118Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0032221Z Ran 1 test in 1.874s 2022-11-23T01:42:00.0032226Z 2022-11-23T01:42:00.0032285Z OK 2022-11-23T01:42:00.0032290Z 2022-11-23T01:42:00.0032371Z Generating XML reports... 2022-11-23T01:42:00.0032789Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013705.xml 2022-11-23T01:42:00.0033064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0033186Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0033459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0033593Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0033777Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxl34ktzm 2022-11-23T01:42:00.0033972Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxl34ktzm/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0034201Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0034206Z 2022-11-23T01:42:00.0034277Z Running tests... 2022-11-23T01:42:00.0034459Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0034617Z test_ddp_comm_hook_sparse_gradients (__main__.DistributedDataParallelTest) 2022-11-23T01:42:00.0034858Z Runs "test_sparse_gradients" unit test with DDP communication hook. We define a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3795 2022-11-23T01:42:00.0035011Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3796 2022-11-23T01:42:00.0035283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0035403Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0035683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0035817Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0036001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1d_tp5d2 2022-11-23T01:42:00.0036186Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1d_tp5d2/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0036351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0036619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0036740Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0037019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0037154Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0037337Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxrsp72c1 2022-11-23T01:42:00.0037573Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxrsp72c1/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0037724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0037791Z ok (1.887s) 2022-11-23T01:42:00.0037796Z 2022-11-23T01:42:00.0037991Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0038064Z Ran 1 test in 1.887s 2022-11-23T01:42:00.0038069Z 2022-11-23T01:42:00.0038128Z OK 2022-11-23T01:42:00.0038133Z 2022-11-23T01:42:00.0038214Z Generating XML reports... 2022-11-23T01:42:00.0038559Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013709.xml 2022-11-23T01:42:00.0038864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0038985Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0039255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0039389Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0039573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk54bnkmu 2022-11-23T01:42:00.0039768Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk54bnkmu/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0039998Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0040003Z 2022-11-23T01:42:00.0040075Z Running tests... 2022-11-23T01:42:00.0040269Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0040423Z test_ddp_invalid_comm_hook_init (__main__.DistributedDataParallelTest) 2022-11-23T01:42:00.0040777Z This unit test makes sure that register_comm_hook properly checks the format ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3840 2022-11-23T01:42:00.0040938Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3841 2022-11-23T01:42:00.0041215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0041337Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0041618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0041752Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0041936Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdez013u0 2022-11-23T01:42:00.0042132Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdez013u0/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0042401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0042511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0042791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0042923Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0043105Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg7r745y8 2022-11-23T01:42:00.0043298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg7r745y8/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0043461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0043623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0043693Z ok (1.872s) 2022-11-23T01:42:00.0043699Z 2022-11-23T01:42:00.0043882Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0043958Z Ran 1 test in 1.872s 2022-11-23T01:42:00.0043963Z 2022-11-23T01:42:00.0044023Z OK 2022-11-23T01:42:00.0044090Z 2022-11-23T01:42:00.0044175Z Generating XML reports... 2022-11-23T01:42:00.0044526Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013712.xml 2022-11-23T01:42:00.0044800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0044926Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0045206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0045338Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0045550Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpizfpjn77 2022-11-23T01:42:00.0045748Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpizfpjn77/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0045979Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0045985Z 2022-11-23T01:42:00.0046056Z Running tests... 2022-11-23T01:42:00.0046249Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0046407Z test_ddp_invalid_comm_hook_return_type (__main__.DistributedDataParallelTest) 2022-11-23T01:42:00.0046655Z This test checks whether return annotation checked properly if defined. It also ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3879 2022-11-23T01:42:00.0046808Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3880 2022-11-23T01:42:00.0047070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0047192Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0047476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0047609Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0047792Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8kizx9ki 2022-11-23T01:42:00.0047986Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8kizx9ki/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0048256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0048378Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0048654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0048777Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0048960Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkdlf8fey 2022-11-23T01:42:00.0049156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkdlf8fey/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0049319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0049486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0049556Z ok (1.876s) 2022-11-23T01:42:00.0049561Z 2022-11-23T01:42:00.0049756Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0049830Z Ran 1 test in 1.876s 2022-11-23T01:42:00.0049834Z 2022-11-23T01:42:00.0049882Z OK 2022-11-23T01:42:00.0049897Z 2022-11-23T01:42:00.0049968Z Generating XML reports... 2022-11-23T01:42:00.0050313Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013716.xml 2022-11-23T01:42:00.0050586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0050745Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0051025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0051160Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0051345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu63czi3t 2022-11-23T01:42:00.0051540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu63czi3t/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0051758Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0051815Z 2022-11-23T01:42:00.0051874Z Running tests... 2022-11-23T01:42:00.0052068Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0052257Z test_find_unused_parameters_when_unused_parameters_empty (__main__.DistributedDataParallelTest) 2022-11-23T01:42:00.0052495Z An empty unused_parameters array does not imply find_unused_parameters = ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3918 2022-11-23T01:42:00.0052649Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3919 2022-11-23T01:42:00.0052922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0053044Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0053324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0053449Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0053631Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6_5_zutg 2022-11-23T01:42:00.0053823Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6_5_zutg/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0053987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0054258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0054379Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0054660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0054794Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0054965Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpapzzrivy 2022-11-23T01:42:00.0055163Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpapzzrivy/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0055325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0055429Z skip: Need at least 2 CUDA devices (1.876s) 2022-11-23T01:42:00.0055437Z 2022-11-23T01:42:00.0055629Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0055703Z Ran 1 test in 1.876s 2022-11-23T01:42:00.0055707Z 2022-11-23T01:42:00.0055777Z OK (skipped=1) 2022-11-23T01:42:00.0055782Z 2022-11-23T01:42:00.0055864Z Generating XML reports... 2022-11-23T01:42:00.0056211Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013719.xml 2022-11-23T01:42:00.0056472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0056596Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0056880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0057016Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0057240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfho6wrd6 2022-11-23T01:42:00.0057438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfho6wrd6/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0057669Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0057674Z 2022-11-23T01:42:00.0057748Z Running tests... 2022-11-23T01:42:00.0057928Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0058180Z test_global_local_unused_params_grad (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3951 2022-11-23T01:42:00.0058367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3952 2022-11-23T01:42:00.0058638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0058766Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0059046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0059180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0059364Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa0zsj12u 2022-11-23T01:42:00.0059558Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa0zsj12u/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0059708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0059981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0060106Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0060385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0060519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0060701Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu3bxfhnv 2022-11-23T01:42:00.0060897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu3bxfhnv/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0061059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0061162Z skip: Need at least 2 CUDA devices (1.874s) 2022-11-23T01:42:00.0061167Z 2022-11-23T01:42:00.0061349Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0061425Z Ran 1 test in 1.874s 2022-11-23T01:42:00.0061431Z 2022-11-23T01:42:00.0061500Z OK (skipped=1) 2022-11-23T01:42:00.0061505Z 2022-11-23T01:42:00.0061590Z Generating XML reports... 2022-11-23T01:42:00.0061940Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013723.xml 2022-11-23T01:42:00.0062212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0062334Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0062610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0062732Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0062915Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_o5jj8xp 2022-11-23T01:42:00.0063108Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_o5jj8xp/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0063338Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0063343Z 2022-11-23T01:42:00.0063416Z Running tests... 2022-11-23T01:42:00.0063648Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0063918Z test_global_local_unused_params_grad_with_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3984 2022-11-23T01:42:00.0064072Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3985 2022-11-23T01:42:00.0064343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0064452Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0064728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0064898Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0065082Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp113mpwdw 2022-11-23T01:42:00.0065283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp113mpwdw/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0065444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0065714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0065837Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0066116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0066237Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0066422Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgrh1aeqo 2022-11-23T01:42:00.0066617Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgrh1aeqo/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0066781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0066884Z skip: Need at least 2 CUDA devices (1.875s) 2022-11-23T01:42:00.0066889Z 2022-11-23T01:42:00.0067081Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0067154Z Ran 1 test in 1.876s 2022-11-23T01:42:00.0067159Z 2022-11-23T01:42:00.0067228Z OK (skipped=1) 2022-11-23T01:42:00.0067233Z 2022-11-23T01:42:00.0067303Z Generating XML reports... 2022-11-23T01:42:00.0067649Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013726.xml 2022-11-23T01:42:00.0067922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0068046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0068324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0068460Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0068644Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpakrn7pg3 2022-11-23T01:42:00.0068838Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpakrn7pg3/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0069063Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0069068Z 2022-11-23T01:42:00.0069127Z Running tests... 2022-11-23T01:42:00.0069319Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0069586Z test_global_local_unused_params_grad_with_static_graph (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4017 2022-11-23T01:42:00.0069742Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4018 2022-11-23T01:42:00.0070062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0070188Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0070467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0070600Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0070782Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppgtuulbz 2022-11-23T01:42:00.0070968Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppgtuulbz/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0071132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0071438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0071561Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0071841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0071972Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0072154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf1r392ns 2022-11-23T01:42:00.0072346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf1r392ns/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0072495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0072660Z skip: Need at least 2 CUDA devices (1.873s) 2022-11-23T01:42:00.0072669Z 2022-11-23T01:42:00.0072869Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0072946Z Ran 1 test in 1.873s 2022-11-23T01:42:00.0072951Z 2022-11-23T01:42:00.0073022Z OK (skipped=1) 2022-11-23T01:42:00.0073026Z 2022-11-23T01:42:00.0073109Z Generating XML reports... 2022-11-23T01:42:00.0073454Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013730.xml 2022-11-23T01:42:00.0073727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0073849Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0074117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0074253Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0074435Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_w1hmva3 2022-11-23T01:42:00.0074632Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_w1hmva3/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0074858Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0074863Z 2022-11-23T01:42:00.0074937Z Running tests... 2022-11-23T01:42:00.0075132Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0075404Z test_gloo_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4050 2022-11-23T01:42:00.0075558Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4051 2022-11-23T01:42:00.0075818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0075941Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0076222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0076357Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0076580Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmo0smwqb 2022-11-23T01:42:00.0076782Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmo0smwqb/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0077052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0077174Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0077441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0077575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0077794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu3ulld3l 2022-11-23T01:42:00.0077990Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu3ulld3l/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0078154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0078316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0078418Z skip: Need at least 2 CUDA devices (1.875s) 2022-11-23T01:42:00.0078424Z 2022-11-23T01:42:00.0078619Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0078693Z Ran 1 test in 1.876s 2022-11-23T01:42:00.0078698Z 2022-11-23T01:42:00.0078755Z OK (skipped=1) 2022-11-23T01:42:00.0078760Z 2022-11-23T01:42:00.0078842Z Generating XML reports... 2022-11-23T01:42:00.0079188Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013733.xml 2022-11-23T01:42:00.0079460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0079581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0079863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0079999Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0080183Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp77h05gj3 2022-11-23T01:42:00.0080363Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp77h05gj3/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0080772Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0080781Z 2022-11-23T01:42:00.0080862Z Running tests... 2022-11-23T01:42:00.0081058Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0081335Z test_gloo_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4083 2022-11-23T01:42:00.0081489Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4084 2022-11-23T01:42:00.0081761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0081885Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0082167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0082289Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0082474Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdph2rs_n 2022-11-23T01:42:00.0082746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0082873Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0083153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0083353Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0083539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcjsjtmby 2022-11-23T01:42:00.0083738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdph2rs_n/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0083929Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcjsjtmby/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0084081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0084242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0084382Z skip: Need at least 2 CUDA devices (1.775s) 2022-11-23T01:42:00.0084387Z 2022-11-23T01:42:00.0084586Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0084661Z Ran 1 test in 1.776s 2022-11-23T01:42:00.0084666Z 2022-11-23T01:42:00.0084738Z OK (skipped=1) 2022-11-23T01:42:00.0084745Z 2022-11-23T01:42:00.0084827Z Generating XML reports... 2022-11-23T01:42:00.0085177Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013737.xml 2022-11-23T01:42:00.0085438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0085562Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0085840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0085975Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0086159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbalz_aqh 2022-11-23T01:42:00.0086351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbalz_aqh/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0086582Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0086587Z 2022-11-23T01:42:00.0086658Z Running tests... 2022-11-23T01:42:00.0086855Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0087089Z test_gloo_backend_2gpu_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4116 2022-11-23T01:42:00.0087244Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4117 2022-11-23T01:42:00.0087515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0087641Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0087919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0088053Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0088240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpebt5wtbm 2022-11-23T01:42:00.0088435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpebt5wtbm/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0088599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0088857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0088981Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0089258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0089396Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0089580Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxw1654ly 2022-11-23T01:42:00.0089805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxw1654ly/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0089969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0090074Z skip: Need at least 4 CUDA devices (1.874s) 2022-11-23T01:42:00.0090079Z 2022-11-23T01:42:00.0090262Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0090336Z Ran 1 test in 1.874s 2022-11-23T01:42:00.0090341Z 2022-11-23T01:42:00.0090415Z OK (skipped=1) 2022-11-23T01:42:00.0090420Z 2022-11-23T01:42:00.0090503Z Generating XML reports... 2022-11-23T01:42:00.0090850Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013740.xml 2022-11-23T01:42:00.0091154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0091276Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0091558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0091693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0091866Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp0yfzhpi 2022-11-23T01:42:00.0092063Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp0yfzhpi/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0092291Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0092296Z 2022-11-23T01:42:00.0092369Z Running tests... 2022-11-23T01:42:00.0092563Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0092807Z test_gloo_backend_4gpu_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4149 2022-11-23T01:42:00.0092963Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4150 2022-11-23T01:42:00.0093236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0093361Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0093628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0093761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0093946Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp17h88we2 2022-11-23T01:42:00.0094143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp17h88we2/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0094304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0094578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0094699Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0094976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0095098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0095281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0n0h0r7z 2022-11-23T01:42:00.0095474Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0n0h0r7z/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0095635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0095740Z skip: Need at least 8 CUDA devices (1.883s) 2022-11-23T01:42:00.0095744Z 2022-11-23T01:42:00.0095938Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0096012Z Ran 1 test in 1.883s 2022-11-23T01:42:00.0096017Z 2022-11-23T01:42:00.0096123Z OK (skipped=1) 2022-11-23T01:42:00.0096128Z 2022-11-23T01:42:00.0096211Z Generating XML reports... 2022-11-23T01:42:00.0096544Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013744.xml 2022-11-23T01:42:00.0096818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0096941Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0097221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0097386Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0097570Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz6wt4ph1 2022-11-23T01:42:00.0097764Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz6wt4ph1/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0097996Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0098001Z 2022-11-23T01:42:00.0098061Z Running tests... 2022-11-23T01:42:00.0098256Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0098498Z test_gloo_backend_cpu_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4182 2022-11-23T01:42:00.0098654Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4183 2022-11-23T01:42:00.0098925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0099049Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0099327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0099463Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0099646Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7idrui0i 2022-11-23T01:42:00.0099829Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7idrui0i/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0100099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0100220Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0100497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0100633Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0100817Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz2h69_rq 2022-11-23T01:42:00.0101011Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz2h69_rq/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0101174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0101336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0101492Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T01:42:00.0101659Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T01:42:00.0101727Z ok (1.874s) 2022-11-23T01:42:00.0101732Z 2022-11-23T01:42:00.0101927Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0102003Z Ran 1 test in 1.874s 2022-11-23T01:42:00.0102008Z 2022-11-23T01:42:00.0102069Z OK 2022-11-23T01:42:00.0102075Z 2022-11-23T01:42:00.0102157Z Generating XML reports... 2022-11-23T01:42:00.0102501Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013747.xml 2022-11-23T01:42:00.0102794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0102918Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0103196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0103330Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0103514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpreik2oc6 2022-11-23T01:42:00.0103709Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpreik2oc6/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0103967Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0103972Z 2022-11-23T01:42:00.0104043Z Running tests... 2022-11-23T01:42:00.0104223Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0104482Z test_gloo_backend_cpu_module_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4221 2022-11-23T01:42:00.0104637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4222 2022-11-23T01:42:00.0104908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0105030Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0105309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0105445Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0105629Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe4sg027w 2022-11-23T01:42:00.0105824Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe4sg027w/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0105974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0106244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0106367Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0106648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0106781Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0106966Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqaudhljl 2022-11-23T01:42:00.0107167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqaudhljl/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0107328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0107499Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T01:42:00.0107658Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T01:42:00.0107725Z ok (1.872s) 2022-11-23T01:42:00.0107730Z 2022-11-23T01:42:00.0107923Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0107999Z Ran 1 test in 1.872s 2022-11-23T01:42:00.0108003Z 2022-11-23T01:42:00.0108064Z OK 2022-11-23T01:42:00.0108068Z 2022-11-23T01:42:00.0108151Z Generating XML reports... 2022-11-23T01:42:00.0108494Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013751.xml 2022-11-23T01:42:00.0108767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0108877Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0109192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0109327Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0109514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp66zpwqtb 2022-11-23T01:42:00.0109706Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp66zpwqtb/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0109933Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0109938Z 2022-11-23T01:42:00.0110010Z Running tests... 2022-11-23T01:42:00.0110205Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0110375Z test_ignored_output (__main__.DistributedDataParallelTest) 2022-11-23T01:42:00.0110588Z Test that the output of a model can be ignored and that there is no ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4260 2022-11-23T01:42:00.0110744Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4261 2022-11-23T01:42:00.0111016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0111140Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0111420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0111554Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0111740Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt8m4hizo 2022-11-23T01:42:00.0111937Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt8m4hizo/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0112088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0112361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0112483Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0112828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0112964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0113147Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0vbdhab5 2022-11-23T01:42:00.0113342Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0vbdhab5/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0113506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0113675Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T01:42:00.0113832Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T01:42:00.0113900Z ok (1.884s) 2022-11-23T01:42:00.0113906Z 2022-11-23T01:42:00.0114101Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0114177Z Ran 1 test in 1.884s 2022-11-23T01:42:00.0114182Z 2022-11-23T01:42:00.0114242Z OK 2022-11-23T01:42:00.0114246Z 2022-11-23T01:42:00.0114328Z Generating XML reports... 2022-11-23T01:42:00.0114675Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013754.xml 2022-11-23T01:42:00.0114946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0115059Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0115339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0115473Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0115706Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8jos4wz_ 2022-11-23T01:42:00.0115903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8jos4wz_/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0116132Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0116137Z 2022-11-23T01:42:00.0116208Z Running tests... 2022-11-23T01:42:00.0116401Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0116566Z test_ignored_output_with_unused_parameters (__main__.DistributedDataParallelTest) 2022-11-23T01:42:00.0116777Z Test that the output of a model can be ignored and that there is no ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4305 2022-11-23T01:42:00.0116957Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4306 2022-11-23T01:42:00.0117229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0117351Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0117633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0117767Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0117949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplkt06_t4 2022-11-23T01:42:00.0118144Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplkt06_t4/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0118416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0118526Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0118807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0118939Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0119122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj2rlevyc 2022-11-23T01:42:00.0119316Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj2rlevyc/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0119478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0119640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0119706Z ok (1.874s) 2022-11-23T01:42:00.0119711Z 2022-11-23T01:42:00.0119895Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0119968Z Ran 1 test in 1.875s 2022-11-23T01:42:00.0119973Z 2022-11-23T01:42:00.0120033Z OK 2022-11-23T01:42:00.0120037Z 2022-11-23T01:42:00.0120120Z Generating XML reports... 2022-11-23T01:42:00.0120466Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013758.xml 2022-11-23T01:42:00.0120960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0121111Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0121394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0121514Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0121698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgzwtqh30 2022-11-23T01:42:00.0121897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgzwtqh30/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0122123Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0122128Z 2022-11-23T01:42:00.0122199Z Running tests... 2022-11-23T01:42:00.0122454Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0122701Z test_ignored_sharded_tensor (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4350 2022-11-23T01:42:00.0122856Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4351 2022-11-23T01:42:00.0123126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0123237Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0123517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0123689Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0123876Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9z10cxj0 2022-11-23T01:42:00.0124072Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9z10cxj0/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0124235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0124508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0124630Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0124914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0125034Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0125219Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx3d1gj3j 2022-11-23T01:42:00.0125415Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx3d1gj3j/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0125579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0125683Z skip: Need at least 2 CUDA devices (1.890s) 2022-11-23T01:42:00.0125688Z 2022-11-23T01:42:00.0125884Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0125959Z Ran 1 test in 1.890s 2022-11-23T01:42:00.0125963Z 2022-11-23T01:42:00.0126033Z OK (skipped=1) 2022-11-23T01:42:00.0126037Z 2022-11-23T01:42:00.0126108Z Generating XML reports... 2022-11-23T01:42:00.0126454Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013801.xml 2022-11-23T01:42:00.0126727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0126850Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0127129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0127263Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0127446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprnf54q9f 2022-11-23T01:42:00.0127641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprnf54q9f/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0127869Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0127874Z 2022-11-23T01:42:00.0127934Z Running tests... 2022-11-23T01:42:00.0128125Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0128369Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4383 2022-11-23T01:42:00.0128523Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4384 2022-11-23T01:42:00.0128821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0128944Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0129225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0129360Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0129542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvi5hngw2 2022-11-23T01:42:00.0129799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0129955Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0130232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0130364Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0130551Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe9oklin9 2022-11-23T01:42:00.0130747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvi5hngw2/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0130939Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe9oklin9/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0131102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0131253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0131665Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T01:42:00.0132081Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T01:42:00.0132487Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T01:42:00.0132899Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T01:42:00.0133303Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T01:42:00.0133712Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T01:42:00.0134149Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T01:42:00.0134548Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T01:42:00.0134941Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T01:42:00.0135372Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T01:42:00.0135767Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T01:42:00.0136167Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T01:42:00.0136236Z ok (1.777s) 2022-11-23T01:42:00.0136241Z 2022-11-23T01:42:00.0136442Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0136517Z Ran 1 test in 1.777s 2022-11-23T01:42:00.0136522Z 2022-11-23T01:42:00.0136582Z OK 2022-11-23T01:42:00.0136587Z 2022-11-23T01:42:00.0136671Z Generating XML reports... 2022-11-23T01:42:00.0137022Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013805.xml 2022-11-23T01:42:00.0137294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0137416Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0137685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0137821Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0138006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3_tv6fwv 2022-11-23T01:42:00.0138201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3_tv6fwv/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0138431Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0138435Z 2022-11-23T01:42:00.0138507Z Running tests... 2022-11-23T01:42:00.0138702Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0138944Z test_save_load_checkpoint (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4416 2022-11-23T01:42:00.0139099Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4417 2022-11-23T01:42:00.0139363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0139486Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0139796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0139933Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0140118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqwn0r8f3 2022-11-23T01:42:00.0140313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqwn0r8f3/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0140475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0140750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0140888Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0141167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0141303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0141485Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6hw4z5tk 2022-11-23T01:42:00.0141678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6hw4z5tk/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0141840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0141943Z skip: Need at least 2 CUDA devices (1.876s) 2022-11-23T01:42:00.0141948Z 2022-11-23T01:42:00.0142144Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0142217Z Ran 1 test in 1.876s 2022-11-23T01:42:00.0142224Z 2022-11-23T01:42:00.0142283Z OK (skipped=1) 2022-11-23T01:42:00.0142287Z 2022-11-23T01:42:00.0142370Z Generating XML reports... 2022-11-23T01:42:00.0142720Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013808.xml 2022-11-23T01:42:00.0142997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0143119Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0143399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0143533Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0143717Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1mxdhvkn 2022-11-23T01:42:00.0143901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1mxdhvkn/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0144132Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0144137Z 2022-11-23T01:42:00.0144209Z Running tests... 2022-11-23T01:42:00.0144402Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0144641Z test_sparse_gradients (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4449 2022-11-23T01:42:00.0144796Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4450 2022-11-23T01:42:00.0145066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0145187Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0145467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0145588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0145775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsvtt5931 2022-11-23T01:42:00.0145968Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsvtt5931/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0146269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0146395Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0146673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0146805Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0146987Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz08mqnb5 2022-11-23T01:42:00.0147180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz08mqnb5/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0147362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0147525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0147591Z ok (1.876s) 2022-11-23T01:42:00.0147596Z 2022-11-23T01:42:00.0147794Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0147869Z Ran 1 test in 1.876s 2022-11-23T01:42:00.0147874Z 2022-11-23T01:42:00.0147933Z OK 2022-11-23T01:42:00.0147937Z 2022-11-23T01:42:00.0148019Z Generating XML reports... 2022-11-23T01:42:00.0148366Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013811.xml 2022-11-23T01:42:00.0148628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0148750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0149030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0149164Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0149348Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_cgy49bi 2022-11-23T01:42:00.0149541Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_cgy49bi/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0149769Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0149774Z 2022-11-23T01:42:00.0149845Z Running tests... 2022-11-23T01:42:00.0150038Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0150274Z test_sparse_gradients_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4494 2022-11-23T01:42:00.0150428Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4495 2022-11-23T01:42:00.0150701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0150822Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0151107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0151245Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0151428Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5qagjeuk 2022-11-23T01:42:00.0151623Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5qagjeuk/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0151772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0152042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0152165Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0152443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0152677Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0152872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2csukwr7 2022-11-23T01:42:00.0153067Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2csukwr7/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0153230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0153296Z ok (1.872s) 2022-11-23T01:42:00.0153301Z 2022-11-23T01:42:00.0153487Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0153565Z Ran 1 test in 1.872s 2022-11-23T01:42:00.0153570Z 2022-11-23T01:42:00.0153665Z OK 2022-11-23T01:42:00.0153670Z 2022-11-23T01:42:00.0153756Z Generating XML reports... 2022-11-23T01:42:00.0154103Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013815.xml 2022-11-23T01:42:00.0154378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0154501Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0154779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0154901Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0155086Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3neeei3r 2022-11-23T01:42:00.0155280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3neeei3r/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0155511Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0155516Z 2022-11-23T01:42:00.0155588Z Running tests... 2022-11-23T01:42:00.0169421Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0169773Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4539 2022-11-23T01:42:00.0169936Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4540 2022-11-23T01:42:00.0170254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0170384Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0170672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0170809Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0171006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwus0tx8l 2022-11-23T01:42:00.0171206Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwus0tx8l/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0171378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0171653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0171764Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0172049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0172187Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0172376Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpteua2p2i 2022-11-23T01:42:00.0172579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpteua2p2i/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0172743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0172847Z skip: Need at least 2 CUDA devices (1.877s) 2022-11-23T01:42:00.0172854Z 2022-11-23T01:42:00.0173255Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0173335Z Ran 1 test in 1.877s 2022-11-23T01:42:00.0173342Z 2022-11-23T01:42:00.0173400Z OK (skipped=1) 2022-11-23T01:42:00.0173404Z 2022-11-23T01:42:00.0173492Z Generating XML reports... 2022-11-23T01:42:00.0173851Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013818.xml 2022-11-23T01:42:00.0174127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0174252Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0174577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0174716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0174905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy1x4z7z4 2022-11-23T01:42:00.0175086Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy1x4z7z4/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0175321Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0175326Z 2022-11-23T01:42:00.0175399Z Running tests... 2022-11-23T01:42:00.0175594Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0175845Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4572 2022-11-23T01:42:00.0176003Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4573 2022-11-23T01:42:00.0176277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0176402Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0176683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0176806Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0176993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp084io7hl 2022-11-23T01:42:00.0177188Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp084io7hl/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0177352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0177624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0177755Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0178032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0178170Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0178355Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm_ygpabb 2022-11-23T01:42:00.0178537Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm_ygpabb/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0178701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0178806Z skip: Need at least 2 CUDA devices (1.873s) 2022-11-23T01:42:00.0178811Z 2022-11-23T01:42:00.0179007Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0179082Z Ran 1 test in 1.873s 2022-11-23T01:42:00.0179089Z 2022-11-23T01:42:00.0179164Z OK (skipped=1) 2022-11-23T01:42:00.0179169Z 2022-11-23T01:42:00.0179253Z Generating XML reports... 2022-11-23T01:42:00.0179604Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013822.xml 2022-11-23T01:42:00.0179892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0180018Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0180300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0180434Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0180619Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe2umq5o0 2022-11-23T01:42:00.0180814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe2umq5o0/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0181080Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0181085Z 2022-11-23T01:42:00.0181158Z Running tests... 2022-11-23T01:42:00.0181353Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0181634Z test_allgather_coalesced (__main__.GlooProcessGroupWithDispatchedCollectivesTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4605 2022-11-23T01:42:00.0181908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0182032Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0182314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0182452Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0182638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1rqjvxe8 2022-11-23T01:42:00.0182837Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1rqjvxe8/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0183004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0183181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:42:00.0183470Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T01:42:00.0184052Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T01:42:00.0184205Z "torch.distributed.all_gather_coalesced will be deprecated. If you must " 2022-11-23T01:42:00.0184276Z ok (1.770s) 2022-11-23T01:42:00.0184282Z 2022-11-23T01:42:00.0184479Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0184555Z Ran 1 test in 1.770s 2022-11-23T01:42:00.0184560Z 2022-11-23T01:42:00.0184621Z OK 2022-11-23T01:42:00.0184626Z 2022-11-23T01:42:00.0184711Z Generating XML reports... 2022-11-23T01:42:00.0185134Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20221123013825.xml 2022-11-23T01:42:00.0185393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0185519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0185801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0185940Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0186130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp84w2g2tu 2022-11-23T01:42:00.0186325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp84w2g2tu/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0186586Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0186593Z 2022-11-23T01:42:00.0186668Z Running tests... 2022-11-23T01:42:00.0186864Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0187146Z test_allreduce_coalesced (__main__.GlooProcessGroupWithDispatchedCollectivesTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4630 2022-11-23T01:42:00.0187418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0187541Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0187849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0187985Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0188172Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp58z5yk3i 2022-11-23T01:42:00.0188367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp58z5yk3i/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0188532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0188696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:42:00.0188996Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T01:42:00.0189560Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T01:42:00.0189716Z "torch.distributed.all_reduce_coalesced will be deprecated. If you must " 2022-11-23T01:42:00.0189784Z ok (1.772s) 2022-11-23T01:42:00.0189791Z 2022-11-23T01:42:00.0189986Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0190062Z Ran 1 test in 1.772s 2022-11-23T01:42:00.0190068Z 2022-11-23T01:42:00.0190128Z OK 2022-11-23T01:42:00.0190133Z 2022-11-23T01:42:00.0190216Z Generating XML reports... 2022-11-23T01:42:00.0190638Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20221123013829.xml 2022-11-23T01:42:00.0190897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0191025Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0191306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0191442Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0191629Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp97tb2_57 2022-11-23T01:42:00.0191822Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp97tb2_57/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0192052Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0192056Z 2022-11-23T01:42:00.0192129Z Running tests... 2022-11-23T01:42:00.0192310Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0192676Z test_collectives (__main__.GlooProcessGroupWithDispatchedCollectivesTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4655 2022-11-23T01:42:00.0192965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0193090Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0193406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0193542Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0193725Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0_efi2d5 2022-11-23T01:42:00.0193919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0_efi2d5/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0194085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0194248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:42:00.0194593Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T01:42:00.0194661Z ok (1.768s) 2022-11-23T01:42:00.0194666Z 2022-11-23T01:42:00.0194861Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0194939Z Ran 1 test in 1.768s 2022-11-23T01:42:00.0194944Z 2022-11-23T01:42:00.0195006Z OK 2022-11-23T01:42:00.0195010Z 2022-11-23T01:42:00.0195096Z Generating XML reports... 2022-11-23T01:42:00.0195510Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20221123013832.xml 2022-11-23T01:42:00.0195776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0195886Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0196162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0196294Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0196478Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp01wf0d_k 2022-11-23T01:42:00.0196671Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp01wf0d_k/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0196896Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0196901Z 2022-11-23T01:42:00.0196968Z Running tests... 2022-11-23T01:42:00.0197160Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0197435Z test_monitored_barrier (__main__.GlooProcessGroupWithDispatchedCollectivesTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4680 2022-11-23T01:42:00.0197706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0197833Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0198113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0198249Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0198437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqic0im1t 2022-11-23T01:42:00.0198634Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqic0im1t/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0198798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0198975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:42:00.0199261Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T01:42:00.0199335Z ok (1.770s) 2022-11-23T01:42:00.0199340Z 2022-11-23T01:42:00.0199535Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0199611Z Ran 1 test in 1.771s 2022-11-23T01:42:00.0199616Z 2022-11-23T01:42:00.0199676Z OK 2022-11-23T01:42:00.0199680Z 2022-11-23T01:42:00.0199763Z Generating XML reports... 2022-11-23T01:42:00.0200211Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20221123013836.xml 2022-11-23T01:42:00.0200488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0200772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0201136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0201272Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0201506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl15h27i1 2022-11-23T01:42:00.0201701Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl15h27i1/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0201934Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0201939Z 2022-11-23T01:42:00.0202015Z Running tests... 2022-11-23T01:42:00.0202211Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0202441Z test_allgather_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4705 2022-11-23T01:42:00.0202585Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4706 2022-11-23T01:42:00.0202740Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4707 2022-11-23T01:42:00.0202890Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4708 2022-11-23T01:42:00.0203167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0203293Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0203578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0203716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0203905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt82yy93e 2022-11-23T01:42:00.0204099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt82yy93e/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0204357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0204482Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0204759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0204896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0205081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph5ee28o5 2022-11-23T01:42:00.0205278Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph5ee28o5/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0205442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0205603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0205874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0205983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0206263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0206401Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0206588Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiuerjv2v 2022-11-23T01:42:00.0206822Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiuerjv2v/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0207097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0207222Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0207504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0207626Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0207810Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprbn47385 2022-11-23T01:42:00.0208033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprbn47385/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0208197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0208362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0208429Z ok (2.086s) 2022-11-23T01:42:00.0208434Z 2022-11-23T01:42:00.0208629Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0208704Z Ran 1 test in 2.087s 2022-11-23T01:42:00.0208709Z 2022-11-23T01:42:00.0208756Z OK 2022-11-23T01:42:00.0208774Z 2022-11-23T01:42:00.0208844Z Generating XML reports... 2022-11-23T01:42:00.0209171Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013839.xml 2022-11-23T01:42:00.0209445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0209570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0209848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0209982Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0210170Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpak6czpx1 2022-11-23T01:42:00.0210365Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpak6czpx1/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0210579Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0210599Z 2022-11-23T01:42:00.0210657Z Running tests... 2022-11-23T01:42:00.0210854Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0211087Z test_allgather_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4772 2022-11-23T01:42:00.0211246Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4773 2022-11-23T01:42:00.0211399Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4774 2022-11-23T01:42:00.0211552Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4775 2022-11-23T01:42:00.0211825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0211948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0212215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0212352Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0212536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk9brsahb 2022-11-23T01:42:00.0212730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk9brsahb/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0212894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0213167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0213321Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0213603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0213725Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0213910Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj9lcwmf3 2022-11-23T01:42:00.0214103Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj9lcwmf3/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0214264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0214567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0214689Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0214972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0215105Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0215290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv3pjc07r 2022-11-23T01:42:00.0215547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0215669Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0215950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0216085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0216272Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9egtuq0y 2022-11-23T01:42:00.0216469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv3pjc07r/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0216665Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9egtuq0y/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0216827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0216977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0217082Z skip: Need at least 2 CUDA devices (1.983s) 2022-11-23T01:42:00.0217087Z 2022-11-23T01:42:00.0217282Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0217357Z Ran 1 test in 1.983s 2022-11-23T01:42:00.0217362Z 2022-11-23T01:42:00.0217436Z OK (skipped=1) 2022-11-23T01:42:00.0217441Z 2022-11-23T01:42:00.0217525Z Generating XML reports... 2022-11-23T01:42:00.0217853Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013843.xml 2022-11-23T01:42:00.0218128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0218251Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0218518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0218653Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0218838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8h62o0pk 2022-11-23T01:42:00.0219036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8h62o0pk/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0219266Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0219271Z 2022-11-23T01:42:00.0219344Z Running tests... 2022-11-23T01:42:00.0219540Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0219803Z test_allgather_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4827 2022-11-23T01:42:00.0219947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4828 2022-11-23T01:42:00.0220101Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4829 2022-11-23T01:42:00.0220252Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4830 2022-11-23T01:42:00.0220526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0220648Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0220959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0221095Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0221284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj51xkhta 2022-11-23T01:42:00.0221481Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj51xkhta/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0221630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0221900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0222025Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0222303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0222441Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0222627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_t9ag0og 2022-11-23T01:42:00.0222818Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_t9ag0og/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0223092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0223216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0223480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0223614Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0223799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprym6zuva 2022-11-23T01:42:00.0223994Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprym6zuva/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0224159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0224321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0224593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0224716Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0224981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0225116Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0225298Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxqr5u4sk 2022-11-23T01:42:00.0225493Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxqr5u4sk/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0225658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0225725Z ok (2.183s) 2022-11-23T01:42:00.0225730Z 2022-11-23T01:42:00.0225928Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0226003Z Ran 1 test in 2.184s 2022-11-23T01:42:00.0226039Z 2022-11-23T01:42:00.0226088Z OK 2022-11-23T01:42:00.0226105Z 2022-11-23T01:42:00.0226176Z Generating XML reports... 2022-11-23T01:42:00.0226504Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013846.xml 2022-11-23T01:42:00.0226780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0226902Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0227183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0227346Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0227533Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc00l22va 2022-11-23T01:42:00.0227727Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc00l22va/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0227944Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0227962Z 2022-11-23T01:42:00.0228023Z Running tests... 2022-11-23T01:42:00.0228216Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0228453Z test_allgather_coalesced_async (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4894 2022-11-23T01:42:00.0228608Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4895 2022-11-23T01:42:00.0228765Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4896 2022-11-23T01:42:00.0228920Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4897 2022-11-23T01:42:00.0229193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0229316Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0229584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0229720Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0229902Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfc18d745 2022-11-23T01:42:00.0230097Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfc18d745/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0230259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0230531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0230658Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0230935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0231059Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0231243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxgcc8dzc 2022-11-23T01:42:00.0231438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxgcc8dzc/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0231709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0231832Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0232111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0232247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0232431Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp52dtxxlf 2022-11-23T01:42:00.0232730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp52dtxxlf/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0232884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0233048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0233332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0233456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0233738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0233903Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0234088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2viaa7ne 2022-11-23T01:42:00.0234285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2viaa7ne/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0234436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0234614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:42:00.0234789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:42:00.0234964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:42:00.0235137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:42:00.0235436Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:42:00.0235737Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:42:00.0236033Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:42:00.0236324Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:42:00.0236896Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T01:42:00.0237035Z "torch.distributed.all_gather_coalesced will be deprecated. If you must " 2022-11-23T01:42:00.0237590Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T01:42:00.0237744Z "torch.distributed.all_gather_coalesced will be deprecated. If you must " 2022-11-23T01:42:00.0238305Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T01:42:00.0238454Z "torch.distributed.all_gather_coalesced will be deprecated. If you must " 2022-11-23T01:42:00.0239003Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T01:42:00.0239155Z "torch.distributed.all_gather_coalesced will be deprecated. If you must " 2022-11-23T01:42:00.0239223Z ok (2.081s) 2022-11-23T01:42:00.0239229Z 2022-11-23T01:42:00.0239457Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0239534Z Ran 1 test in 2.081s 2022-11-23T01:42:00.0239539Z 2022-11-23T01:42:00.0239601Z OK 2022-11-23T01:42:00.0239606Z 2022-11-23T01:42:00.0239677Z Generating XML reports... 2022-11-23T01:42:00.0240004Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013850.xml 2022-11-23T01:42:00.0240278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0240403Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0240867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0241074Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0241261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ixnemn3 2022-11-23T01:42:00.0241461Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ixnemn3/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0241682Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0241702Z 2022-11-23T01:42:00.0241762Z Running tests... 2022-11-23T01:42:00.0241958Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0242197Z test_allgather_coalesced_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4961 2022-11-23T01:42:00.0242354Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4962 2022-11-23T01:42:00.0242511Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4963 2022-11-23T01:42:00.0242664Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4964 2022-11-23T01:42:00.0242941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0243068Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0243335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0243470Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0243654Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp92ft33i4 2022-11-23T01:42:00.0243851Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp92ft33i4/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0244015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0244290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0244413Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0244694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0244832Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0245002Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp67l3v0z0 2022-11-23T01:42:00.0245197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp67l3v0z0/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0245468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0245595Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0245876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0246012Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0246238Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpao8upb04 2022-11-23T01:42:00.0246435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpao8upb04/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0246586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0246856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0246979Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0247261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0247427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0247608Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6g81w38w 2022-11-23T01:42:00.0247802Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6g81w38w/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0247968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0248130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0248679Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T01:42:00.0248833Z "torch.distributed.all_gather_coalesced will be deprecated. If you must " 2022-11-23T01:42:00.0249391Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T01:42:00.0249542Z "torch.distributed.all_gather_coalesced will be deprecated. If you must " 2022-11-23T01:42:00.0250097Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T01:42:00.0250243Z "torch.distributed.all_gather_coalesced will be deprecated. If you must " 2022-11-23T01:42:00.0250795Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T01:42:00.0250943Z "torch.distributed.all_gather_coalesced will be deprecated. If you must " 2022-11-23T01:42:00.0251016Z ok (2.094s) 2022-11-23T01:42:00.0251021Z 2022-11-23T01:42:00.0251218Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0251297Z Ran 1 test in 2.095s 2022-11-23T01:42:00.0251302Z 2022-11-23T01:42:00.0251350Z OK 2022-11-23T01:42:00.0251355Z 2022-11-23T01:42:00.0251437Z Generating XML reports... 2022-11-23T01:42:00.0251769Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013854.xml 2022-11-23T01:42:00.0252043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0252169Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0252446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0252583Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0252769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcpohoogx 2022-11-23T01:42:00.0252994Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcpohoogx/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0253214Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0253219Z 2022-11-23T01:42:00.0253293Z Running tests... 2022-11-23T01:42:00.0253489Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0253728Z test_allgather_noncontiguous_input (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5028 2022-11-23T01:42:00.0253885Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5029 2022-11-23T01:42:00.0254065Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5030 2022-11-23T01:42:00.0254216Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5031 2022-11-23T01:42:00.0254493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0254604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0254887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0255023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0255209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpygthy1c_ 2022-11-23T01:42:00.0255404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpygthy1c_/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0255567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0255841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0255966Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0256247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0256369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0256554Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfo0pkp6u 2022-11-23T01:42:00.0256748Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfo0pkp6u/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0256911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0257181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0257305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0257585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0257721Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0257892Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwn9934zn 2022-11-23T01:42:00.0258087Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwn9934zn/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0258249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0258523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0258647Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0258928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0259062Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0259247Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppuvt20j9 2022-11-23T01:42:00.0259477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppuvt20j9/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0259627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0259695Z ok (2.083s) 2022-11-23T01:42:00.0259700Z 2022-11-23T01:42:00.0259897Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0259975Z Ran 1 test in 2.083s 2022-11-23T01:42:00.0259980Z 2022-11-23T01:42:00.0260041Z OK 2022-11-23T01:42:00.0260046Z 2022-11-23T01:42:00.0260131Z Generating XML reports... 2022-11-23T01:42:00.0260456Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013857.xml 2022-11-23T01:42:00.0260757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0260867Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0261151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0261287Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0261471Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7zw5wlaa 2022-11-23T01:42:00.0261664Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7zw5wlaa/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0261895Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0261900Z 2022-11-23T01:42:00.0261972Z Running tests... 2022-11-23T01:42:00.0262170Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0262393Z test_allgather_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5095 2022-11-23T01:42:00.0262536Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5096 2022-11-23T01:42:00.0262690Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5097 2022-11-23T01:42:00.0262843Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5098 2022-11-23T01:42:00.0263122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0263246Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0263526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0263661Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0263848Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp8cvl61m 2022-11-23T01:42:00.0264030Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp8cvl61m/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0264195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0264467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0264591Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0264871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0265006Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0265191Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprubx62ej 2022-11-23T01:42:00.0265389Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprubx62ej/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0265662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0265818Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0266102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0266239Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0266424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjj2e_z9a 2022-11-23T01:42:00.0266621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjj2e_z9a/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0266890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0267042Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0267323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0267460Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0267634Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa97u6ezw 2022-11-23T01:42:00.0267827Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa97u6ezw/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0267990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0268152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0268314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0268382Z ok (3.492s) 2022-11-23T01:42:00.0268389Z 2022-11-23T01:42:00.0268590Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0268666Z Ran 1 test in 3.492s 2022-11-23T01:42:00.0268671Z 2022-11-23T01:42:00.0268718Z OK 2022-11-23T01:42:00.0268722Z 2022-11-23T01:42:00.0268805Z Generating XML reports... 2022-11-23T01:42:00.0269134Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013901.xml 2022-11-23T01:42:00.0269406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0269529Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0269807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0269943Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0270127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9iys7ec2 2022-11-23T01:42:00.0270310Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9iys7ec2/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0270538Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0270543Z 2022-11-23T01:42:00.0270616Z Running tests... 2022-11-23T01:42:00.0270814Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0271044Z test_allgather_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5186 2022-11-23T01:42:00.0271200Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5187 2022-11-23T01:42:00.0271352Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5188 2022-11-23T01:42:00.0271504Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5189 2022-11-23T01:42:00.0271774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0271885Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0272165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0272327Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0272515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqng525jq 2022-11-23T01:42:00.0272798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqng525jq/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0272969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0273246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0273369Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0273719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0273842Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0274029Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ntxs6a_ 2022-11-23T01:42:00.0274223Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ntxs6a_/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0274385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0274655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0274780Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0275061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0275199Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0275370Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpofwu67ho 2022-11-23T01:42:00.0275567Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpofwu67ho/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0275840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0275964Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0276245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0276379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0276563Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr1ayt7of 2022-11-23T01:42:00.0276758Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr1ayt7of/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0276926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0277076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0277185Z skip: Need at least 2 CUDA devices (1.979s) 2022-11-23T01:42:00.0277191Z 2022-11-23T01:42:00.0277387Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0277464Z Ran 1 test in 1.979s 2022-11-23T01:42:00.0277468Z 2022-11-23T01:42:00.0277540Z OK (skipped=1) 2022-11-23T01:42:00.0277545Z 2022-11-23T01:42:00.0277629Z Generating XML reports... 2022-11-23T01:42:00.0277952Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013906.xml 2022-11-23T01:42:00.0278229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0278342Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0278620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0278756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0278973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjxs5qk3j 2022-11-23T01:42:00.0279171Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjxs5qk3j/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0279403Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0279408Z 2022-11-23T01:42:00.0279481Z Running tests... 2022-11-23T01:42:00.0279678Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0279903Z test_allreduce_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5241 2022-11-23T01:42:00.0280073Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5242 2022-11-23T01:42:00.0280228Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5243 2022-11-23T01:42:00.0280382Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5244 2022-11-23T01:42:00.0280809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0280989Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0281271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0281407Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0281595Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq69lsqem 2022-11-23T01:42:00.0281775Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq69lsqem/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0282047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0282170Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0282452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0282589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0282778Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk_1ftaw9 2022-11-23T01:42:00.0282972Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk_1ftaw9/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0283133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0283297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0283558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0283684Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0283968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0284103Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0284290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_u575v1n 2022-11-23T01:42:00.0284484Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_u575v1n/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0284756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0284880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0285160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0285283Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0285468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc0j3s2yo 2022-11-23T01:42:00.0285709Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc0j3s2yo/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0285872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0286035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0286103Z ok (2.180s) 2022-11-23T01:42:00.0286108Z 2022-11-23T01:42:00.0286306Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0286381Z Ran 1 test in 2.180s 2022-11-23T01:42:00.0286386Z 2022-11-23T01:42:00.0286433Z OK 2022-11-23T01:42:00.0286438Z 2022-11-23T01:42:00.0286561Z Generating XML reports... 2022-11-23T01:42:00.0286887Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013910.xml 2022-11-23T01:42:00.0287156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0287282Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0287562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0287697Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0287883Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpktbqsbv_ 2022-11-23T01:42:00.0288066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpktbqsbv_/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0288294Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0288302Z 2022-11-23T01:42:00.0288375Z Running tests... 2022-11-23T01:42:00.0288567Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0288799Z test_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5308 2022-11-23T01:42:00.0288955Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5309 2022-11-23T01:42:00.0289111Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5310 2022-11-23T01:42:00.0289263Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5311 2022-11-23T01:42:00.0289538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0289648Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0289931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0290069Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0290254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeplha6vp 2022-11-23T01:42:00.0290453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeplha6vp/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0290728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0290854Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0291133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0291267Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0291438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1axrpmfg 2022-11-23T01:42:00.0291636Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1axrpmfg/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0291798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0291989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0292262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0292386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0292664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0292798Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0292968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp81l5p_ws 2022-11-23T01:42:00.0293235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0293386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0293662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0293797Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0293983Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp92si32km 2022-11-23T01:42:00.0294180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp81l5p_ws/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0294370Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp92si32km/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0294536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0294685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0294792Z skip: Need at least 2 CUDA devices (2.081s) 2022-11-23T01:42:00.0294798Z 2022-11-23T01:42:00.0294994Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0295070Z Ran 1 test in 2.081s 2022-11-23T01:42:00.0295075Z 2022-11-23T01:42:00.0295153Z OK (skipped=1) 2022-11-23T01:42:00.0295158Z 2022-11-23T01:42:00.0295242Z Generating XML reports... 2022-11-23T01:42:00.0295566Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013914.xml 2022-11-23T01:42:00.0295840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0295951Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0296231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0296366Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0296553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpue8aq7vk 2022-11-23T01:42:00.0296750Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpue8aq7vk/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0296983Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0296988Z 2022-11-23T01:42:00.0297061Z Running tests... 2022-11-23T01:42:00.0297258Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0297502Z test_allreduce_basics_cuda_using_work_api (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5363 2022-11-23T01:42:00.0297644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5364 2022-11-23T01:42:00.0297797Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5365 2022-11-23T01:42:00.0297951Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5366 2022-11-23T01:42:00.0298223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0298347Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0298661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0298798Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0298982Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8j7e6wlv 2022-11-23T01:42:00.0299165Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8j7e6wlv/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0299328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0299600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0299752Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0300030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0300168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0300350Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpitll_5vc 2022-11-23T01:42:00.0300544Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpitll_5vc/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0300707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0300964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0301087Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0301368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0301502Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0301687Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9eeiblx_ 2022-11-23T01:42:00.0301881Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9eeiblx_/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0302155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0302278Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0302559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0302680Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0302862Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpverk_434 2022-11-23T01:42:00.0303055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpverk_434/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0303217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0303382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0303486Z skip: Need at least 2 CUDA devices (1.981s) 2022-11-23T01:42:00.0303491Z 2022-11-23T01:42:00.0303687Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0303762Z Ran 1 test in 1.981s 2022-11-23T01:42:00.0303767Z 2022-11-23T01:42:00.0303825Z OK (skipped=1) 2022-11-23T01:42:00.0303829Z 2022-11-23T01:42:00.0303914Z Generating XML reports... 2022-11-23T01:42:00.0304240Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013917.xml 2022-11-23T01:42:00.0304516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0304639Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0304945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0305084Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0305269Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1go36n2c 2022-11-23T01:42:00.0305467Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1go36n2c/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0305682Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0305687Z 2022-11-23T01:42:00.0305759Z Running tests... 2022-11-23T01:42:00.0305955Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0306227Z test_allreduce_basics_using_work_api (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5418 2022-11-23T01:42:00.0306381Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5419 2022-11-23T01:42:00.0306537Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5420 2022-11-23T01:42:00.0306688Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5421 2022-11-23T01:42:00.0306959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0307070Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0307351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0307486Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0307674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpho7fm1fk 2022-11-23T01:42:00.0307871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpho7fm1fk/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0308145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0308269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0308547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0308681Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0308852Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbemaqltv 2022-11-23T01:42:00.0309049Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbemaqltv/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0309213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0309378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0309651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0309777Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0310058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0310191Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0310362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprihdgtze 2022-11-23T01:42:00.0310557Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprihdgtze/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0310825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0310949Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0311229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0311410Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0311597Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq3as7733 2022-11-23T01:42:00.0311788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq3as7733/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0311951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0312101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0312170Z ok (2.179s) 2022-11-23T01:42:00.0312175Z 2022-11-23T01:42:00.0312373Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0312478Z Ran 1 test in 2.179s 2022-11-23T01:42:00.0312483Z 2022-11-23T01:42:00.0312544Z OK 2022-11-23T01:42:00.0312549Z 2022-11-23T01:42:00.0312712Z Generating XML reports... 2022-11-23T01:42:00.0313045Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013921.xml 2022-11-23T01:42:00.0313317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0313428Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0313711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0313846Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0314034Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptj3p5eh_ 2022-11-23T01:42:00.0314231Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptj3p5eh_/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0314466Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0314471Z 2022-11-23T01:42:00.0314545Z Running tests... 2022-11-23T01:42:00.0314743Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0314968Z test_allreduce_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5485 2022-11-23T01:42:00.0315110Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5486 2022-11-23T01:42:00.0315264Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5487 2022-11-23T01:42:00.0315415Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5488 2022-11-23T01:42:00.0315688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0315814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0316094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0316232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0316420Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkpezn2xo 2022-11-23T01:42:00.0316604Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkpezn2xo/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0316768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0317037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0317161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0317442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0317580Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0317772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgh2bb2ip 2022-11-23T01:42:00.0318012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgh2bb2ip/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0318282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0318392Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0318669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0318801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0318983Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp977wwu8b 2022-11-23T01:42:00.0319209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp977wwu8b/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0319484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0319610Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0319895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0320030Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0320199Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9izdc2i9 2022-11-23T01:42:00.0320394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9izdc2i9/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0320558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0320912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0321078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0321148Z ok (2.082s) 2022-11-23T01:42:00.0321153Z 2022-11-23T01:42:00.0321354Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0321432Z Ran 1 test in 2.082s 2022-11-23T01:42:00.0321437Z 2022-11-23T01:42:00.0321484Z OK 2022-11-23T01:42:00.0321489Z 2022-11-23T01:42:00.0321574Z Generating XML reports... 2022-11-23T01:42:00.0321902Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013925.xml 2022-11-23T01:42:00.0322177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0322303Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0322585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0322722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0322909Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpph7a7tb9 2022-11-23T01:42:00.0323108Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpph7a7tb9/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0323324Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0323329Z 2022-11-23T01:42:00.0323401Z Running tests... 2022-11-23T01:42:00.0323598Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0323834Z test_allreduce_coalesced_async (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5552 2022-11-23T01:42:00.0323990Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5553 2022-11-23T01:42:00.0324144Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5554 2022-11-23T01:42:00.0324295Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5555 2022-11-23T01:42:00.0324621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0324733Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0325011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0325148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0325334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnoirpa_4 2022-11-23T01:42:00.0325533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnoirpa_4/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0325806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0325968Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0326249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0326384Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0326558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa4gk0hfc 2022-11-23T01:42:00.0326754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa4gk0hfc/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0326917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0327080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0327353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0327480Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0327759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0327898Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0328069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_oyrxyia 2022-11-23T01:42:00.0328263Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_oyrxyia/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0328534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0328657Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0328937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0329073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0329256Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplc6__k8z 2022-11-23T01:42:00.0329448Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplc6__k8z/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0329615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0329764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0329941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:42:00.0330117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T01:42:00.0330291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:42:00.0330465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T01:42:00.0330767Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:42:00.0331092Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:42:00.0331391Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:42:00.0331686Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T01:42:00.0332245Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T01:42:00.0332430Z "torch.distributed.all_reduce_coalesced will be deprecated. If you must " 2022-11-23T01:42:00.0332986Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T01:42:00.0333137Z "torch.distributed.all_reduce_coalesced will be deprecated. If you must " 2022-11-23T01:42:00.0333698Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T01:42:00.0333845Z "torch.distributed.all_reduce_coalesced will be deprecated. If you must " 2022-11-23T01:42:00.0334393Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T01:42:00.0334544Z "torch.distributed.all_reduce_coalesced will be deprecated. If you must " 2022-11-23T01:42:00.0334614Z ok (2.081s) 2022-11-23T01:42:00.0334620Z 2022-11-23T01:42:00.0334818Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0334892Z Ran 1 test in 2.081s 2022-11-23T01:42:00.0334897Z 2022-11-23T01:42:00.0334945Z OK 2022-11-23T01:42:00.0334964Z 2022-11-23T01:42:00.0335034Z Generating XML reports... 2022-11-23T01:42:00.0335356Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013928.xml 2022-11-23T01:42:00.0335628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0335752Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0336035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0336174Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0336363Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4szeqz3b 2022-11-23T01:42:00.0336562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4szeqz3b/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0336778Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0336796Z 2022-11-23T01:42:00.0336856Z Running tests... 2022-11-23T01:42:00.0337050Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0337286Z test_allreduce_coalesced_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5619 2022-11-23T01:42:00.0337444Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5620 2022-11-23T01:42:00.0337597Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5621 2022-11-23T01:42:00.0337749Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5622 2022-11-23T01:42:00.0338049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0338175Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0338446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0338583Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0338769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuojsiz7h 2022-11-23T01:42:00.0338966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuojsiz7h/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0339158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0339432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0339559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0339839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0339960Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0340145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptaqwkim2 2022-11-23T01:42:00.0340341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptaqwkim2/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0340612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0340736Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0341013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0341151Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0341337Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu7_n8ofs 2022-11-23T01:42:00.0341531Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu7_n8ofs/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0341681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0341840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0342112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0342236Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0342514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0342648Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0342834Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8byon919 2022-11-23T01:42:00.0343029Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8byon919/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0343180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0343248Z ok (2.083s) 2022-11-23T01:42:00.0343253Z 2022-11-23T01:42:00.0343449Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0343525Z Ran 1 test in 2.083s 2022-11-23T01:42:00.0343529Z 2022-11-23T01:42:00.0343591Z OK 2022-11-23T01:42:00.0343595Z 2022-11-23T01:42:00.0343683Z Generating XML reports... 2022-11-23T01:42:00.0344007Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013932.xml 2022-11-23T01:42:00.0344282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0344439Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0344709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0344845Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0345032Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvnm1rv26 2022-11-23T01:42:00.0345229Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvnm1rv26/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0345458Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0345492Z 2022-11-23T01:42:00.0345566Z Running tests... 2022-11-23T01:42:00.0345762Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0346002Z test_allreduce_coalesced_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5686 2022-11-23T01:42:00.0346144Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5687 2022-11-23T01:42:00.0346297Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5688 2022-11-23T01:42:00.0346450Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5689 2022-11-23T01:42:00.0346723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0346846Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0347126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0347263Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0347448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplnu_tphm 2022-11-23T01:42:00.0347649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplnu_tphm/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0347800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0348070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0348192Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0348474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0348608Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0348794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdfw54l1u 2022-11-23T01:42:00.0348988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdfw54l1u/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0349153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0349413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0349536Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0349816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0349951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0350134Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp55_84pkm 2022-11-23T01:42:00.0350327Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp55_84pkm/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0350602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0350724Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0351036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0351158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0351342Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppa50agvz 2022-11-23T01:42:00.0351535Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppa50agvz/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0351696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0351858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0351955Z ok (2.185s) 2022-11-23T01:42:00.0351961Z 2022-11-23T01:42:00.0352159Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0352235Z Ran 1 test in 2.185s 2022-11-23T01:42:00.0352240Z 2022-11-23T01:42:00.0352288Z OK 2022-11-23T01:42:00.0352305Z 2022-11-23T01:42:00.0352377Z Generating XML reports... 2022-11-23T01:42:00.0352775Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013936.xml 2022-11-23T01:42:00.0353053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0353177Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0353458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0353595Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0353783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2pm5bq3z 2022-11-23T01:42:00.0353978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2pm5bq3z/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0354198Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0354203Z 2022-11-23T01:42:00.0354277Z Running tests... 2022-11-23T01:42:00.0354473Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0354717Z test_allreduce_coalesced_checks_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5753 2022-11-23T01:42:00.0354872Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5754 2022-11-23T01:42:00.0355027Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5755 2022-11-23T01:42:00.0355179Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5756 2022-11-23T01:42:00.0355459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0355587Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0355855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0355991Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0356177Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkgx4dsix 2022-11-23T01:42:00.0356451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0356574Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0356853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0356992Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0357175Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwn0kznq0 2022-11-23T01:42:00.0357409Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkgx4dsix/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0357602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwn0kznq0/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0357766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0357928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0358202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0358330Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0358611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0358779Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0358963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy7ay7akc 2022-11-23T01:42:00.0359145Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy7ay7akc/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0359307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0359578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0359702Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0359984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0360120Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0360308Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc2kwcb_8 2022-11-23T01:42:00.0360503Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc2kwcb_8/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0360819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0360969Z skip: Need at least 1 CUDA device (2.080s) 2022-11-23T01:42:00.0360976Z 2022-11-23T01:42:00.0361184Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0361260Z Ran 1 test in 2.080s 2022-11-23T01:42:00.0361266Z 2022-11-23T01:42:00.0361339Z OK (skipped=1) 2022-11-23T01:42:00.0361343Z 2022-11-23T01:42:00.0361430Z Generating XML reports... 2022-11-23T01:42:00.0361757Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013940.xml 2022-11-23T01:42:00.0362031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0362157Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0362423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0362564Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0362750Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4420b9e6 2022-11-23T01:42:00.0362943Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4420b9e6/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0363173Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0363178Z 2022-11-23T01:42:00.0363252Z Running tests... 2022-11-23T01:42:00.0363450Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0363688Z test_allreduce_coalesced_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5808 2022-11-23T01:42:00.0363832Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5809 2022-11-23T01:42:00.0363987Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5810 2022-11-23T01:42:00.0364191Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5811 2022-11-23T01:42:00.0364466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0364591Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0364874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0365010Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0365196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ra9ley5 2022-11-23T01:42:00.0365430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ra9ley5/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0365581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0365857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0365979Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0366259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0366395Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0366582Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnblxmo8m 2022-11-23T01:42:00.0366780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnblxmo8m/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0367055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0367178Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0367446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0367581Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0367765Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp86d9efnb 2022-11-23T01:42:00.0367960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp86d9efnb/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0368234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0368360Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0368639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0368776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0368948Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq6iol0pa 2022-11-23T01:42:00.0369146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq6iol0pa/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0369310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0369470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0369632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0369700Z ok (2.880s) 2022-11-23T01:42:00.0369706Z 2022-11-23T01:42:00.0369903Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0369980Z Ran 1 test in 2.880s 2022-11-23T01:42:00.0369987Z 2022-11-23T01:42:00.0370036Z OK 2022-11-23T01:42:00.0370055Z 2022-11-23T01:42:00.0370126Z Generating XML reports... 2022-11-23T01:42:00.0370447Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013943.xml 2022-11-23T01:42:00.0370751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0370876Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0371158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0371292Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0371476Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiuygzk90 2022-11-23T01:42:00.0371672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiuygzk90/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0371917Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0371936Z 2022-11-23T01:42:00.0371996Z Running tests... 2022-11-23T01:42:00.0372190Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0372420Z test_allreduce_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5899 2022-11-23T01:42:00.0372575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5900 2022-11-23T01:42:00.0372728Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5901 2022-11-23T01:42:00.0372880Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5902 2022-11-23T01:42:00.0373153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0373278Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0373548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0373683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0373870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvic1f7dv 2022-11-23T01:42:00.0374068Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvic1f7dv/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0374232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0374506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0374629Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0374909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0375033Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0375217Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphzt1c08r 2022-11-23T01:42:00.0375413Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphzt1c08r/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0375686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0375810Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0376090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0376224Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0376407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjuska6ua 2022-11-23T01:42:00.0376603Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjuska6ua/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0376863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0376987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0377327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0377461Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0377645Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqshquo_4 2022-11-23T01:42:00.0377842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqshquo_4/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0378008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0378172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0378354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0378424Z ok (2.682s) 2022-11-23T01:42:00.0378429Z 2022-11-23T01:42:00.0378628Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0378706Z Ran 1 test in 2.683s 2022-11-23T01:42:00.0378711Z 2022-11-23T01:42:00.0378773Z OK 2022-11-23T01:42:00.0378778Z 2022-11-23T01:42:00.0378863Z Generating XML reports... 2022-11-23T01:42:00.0379194Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013948.xml 2022-11-23T01:42:00.0379465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0379588Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0379855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0379993Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0380178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5re1gaca 2022-11-23T01:42:00.0380378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5re1gaca/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0380609Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0380614Z 2022-11-23T01:42:00.0380687Z Running tests... 2022-11-23T01:42:00.0380880Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0381111Z test_allreduce_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5990 2022-11-23T01:42:00.0381254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5991 2022-11-23T01:42:00.0381408Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5992 2022-11-23T01:42:00.0381563Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5993 2022-11-23T01:42:00.0381834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0381960Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0382240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0382377Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0382563Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd8w64ovn 2022-11-23T01:42:00.0382759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd8w64ovn/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0382908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0383185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0383308Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0383617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0383753Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0383941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaco_hgi7 2022-11-23T01:42:00.0384136Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaco_hgi7/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0384298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0384556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0384712Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0384992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0385129Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0385317Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxbf_cghn 2022-11-23T01:42:00.0385511Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxbf_cghn/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0385673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0385947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0386071Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0386339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0386475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0386661Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphtv65xx_ 2022-11-23T01:42:00.0386858Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphtv65xx_/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0387020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0387124Z skip: Need at least 2 CUDA devices (2.078s) 2022-11-23T01:42:00.0387130Z 2022-11-23T01:42:00.0387325Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0387400Z Ran 1 test in 2.078s 2022-11-23T01:42:00.0387405Z 2022-11-23T01:42:00.0387463Z OK (skipped=1) 2022-11-23T01:42:00.0387481Z 2022-11-23T01:42:00.0387552Z Generating XML reports... 2022-11-23T01:42:00.0387878Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013952.xml 2022-11-23T01:42:00.0388154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0388279Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0388559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0388695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0388878Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ojvfbdz 2022-11-23T01:42:00.0389074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ojvfbdz/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0389291Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0389310Z 2022-11-23T01:42:00.0389369Z Running tests... 2022-11-23T01:42:00.0389563Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0389793Z test_barrier_implies_wait (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6045 2022-11-23T01:42:00.0389947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6046 2022-11-23T01:42:00.0390133Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6047 2022-11-23T01:42:00.0390287Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6048 2022-11-23T01:42:00.0390558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0390686Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0390955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0391091Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0391307Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphq3yhim9 2022-11-23T01:42:00.0391502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphq3yhim9/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0391777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0391900Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0392180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0392315Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0392486Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5smkpv3u 2022-11-23T01:42:00.0392748Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5smkpv3u/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0392916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0393078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0393360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0393485Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0393771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0393906Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0394088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz_zf5sy_ 2022-11-23T01:42:00.0394268Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz_zf5sy_/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0394537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0394665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0394945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0395081Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0395265Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj4e3dqp1 2022-11-23T01:42:00.0395459Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj4e3dqp1/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0395624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0395774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0395841Z ok (2.180s) 2022-11-23T01:42:00.0395846Z 2022-11-23T01:42:00.0396044Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0396120Z Ran 1 test in 2.180s 2022-11-23T01:42:00.0396124Z 2022-11-23T01:42:00.0396186Z OK 2022-11-23T01:42:00.0396191Z 2022-11-23T01:42:00.0396274Z Generating XML reports... 2022-11-23T01:42:00.0396657Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013956.xml 2022-11-23T01:42:00.0396930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0397053Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0397322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0397457Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0397639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5fl_58v3 2022-11-23T01:42:00.0397863Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5fl_58v3/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0398093Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0398098Z 2022-11-23T01:42:00.0398172Z Running tests... 2022-11-23T01:42:00.0398367Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0398596Z test_broadcast_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6112 2022-11-23T01:42:00.0398737Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6113 2022-11-23T01:42:00.0398892Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6114 2022-11-23T01:42:00.0399045Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6115 2022-11-23T01:42:00.0399321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0399449Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0399729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0399867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0400051Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpchxg2x8f 2022-11-23T01:42:00.0400246Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpchxg2x8f/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0400397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0400870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0401013Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0401298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0401432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0401618Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpptidwnvx 2022-11-23T01:42:00.0401817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpptidwnvx/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0401982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0402239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0402366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0402649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0402783Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0402973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkkulz_pm 2022-11-23T01:42:00.0403169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkkulz_pm/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0403404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0403676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0403799Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0404066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0404204Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0404390Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpklq7dndl 2022-11-23T01:42:00.0404624Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpklq7dndl/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0404787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0404856Z ok (2.185s) 2022-11-23T01:42:00.0404861Z 2022-11-23T01:42:00.0405062Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0405137Z Ran 1 test in 2.185s 2022-11-23T01:42:00.0405142Z 2022-11-23T01:42:00.0405190Z OK 2022-11-23T01:42:00.0405208Z 2022-11-23T01:42:00.0405278Z Generating XML reports... 2022-11-23T01:42:00.0405599Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014000.xml 2022-11-23T01:42:00.0405873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0405995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0406277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0406412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0406599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqg00wsl7 2022-11-23T01:42:00.0406795Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqg00wsl7/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0407009Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0407014Z 2022-11-23T01:42:00.0407086Z Running tests... 2022-11-23T01:42:00.0407281Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0407511Z test_broadcast_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6179 2022-11-23T01:42:00.0407666Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6180 2022-11-23T01:42:00.0407821Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6181 2022-11-23T01:42:00.0407973Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6182 2022-11-23T01:42:00.0408247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0408359Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0408641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0408776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0408958Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_sy1io9s 2022-11-23T01:42:00.0409151Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_sy1io9s/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0409317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0409590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0409715Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0410027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0410150Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0410336Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6v0zsfhy 2022-11-23T01:42:00.0410607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0410731Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0411013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0411176Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0411361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9x7m9e3n 2022-11-23T01:42:00.0411558Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6v0zsfhy/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0411735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9x7m9e3n/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0411901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0412060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0412331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0412454Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0412737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0412874Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0413058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzxtjz_9b 2022-11-23T01:42:00.0413252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzxtjz_9b/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0413400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0413503Z skip: Need at least 2 CUDA devices (1.980s) 2022-11-23T01:42:00.0413508Z 2022-11-23T01:42:00.0413702Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0413778Z Ran 1 test in 1.980s 2022-11-23T01:42:00.0413783Z 2022-11-23T01:42:00.0413855Z OK (skipped=1) 2022-11-23T01:42:00.0413859Z 2022-11-23T01:42:00.0413943Z Generating XML reports... 2022-11-23T01:42:00.0414269Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014004.xml 2022-11-23T01:42:00.0414541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0414666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0414932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0415068Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0415253Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjc3g8en9 2022-11-23T01:42:00.0415452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjc3g8en9/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0415680Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0415688Z 2022-11-23T01:42:00.0415761Z Running tests... 2022-11-23T01:42:00.0415955Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0416180Z test_broadcast_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6234 2022-11-23T01:42:00.0416348Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6235 2022-11-23T01:42:00.0416503Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6236 2022-11-23T01:42:00.0416654Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6237 2022-11-23T01:42:00.0416926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0417051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0417332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0417496Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0417683Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz984xr80 2022-11-23T01:42:00.0417881Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz984xr80/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0418031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0418302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0418425Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0418705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0418839Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0419025Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp8504zbe 2022-11-23T01:42:00.0419300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0419423Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0419693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0419827Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0420013Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprlywhumj 2022-11-23T01:42:00.0420209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp8504zbe/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0420401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprlywhumj/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0420672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0420798Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0421080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0421216Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0421389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ocz8nju 2022-11-23T01:42:00.0421584Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ocz8nju/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0421746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0421906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0422068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0422138Z ok (2.081s) 2022-11-23T01:42:00.0422143Z 2022-11-23T01:42:00.0422338Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0422412Z Ran 1 test in 2.082s 2022-11-23T01:42:00.0422417Z 2022-11-23T01:42:00.0422465Z OK 2022-11-23T01:42:00.0422482Z 2022-11-23T01:42:00.0422581Z Generating XML reports... 2022-11-23T01:42:00.0422908Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014007.xml 2022-11-23T01:42:00.0423181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0423303Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0423581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0423717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0423935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsr8_gu7i 2022-11-23T01:42:00.0424131Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsr8_gu7i/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0424349Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0424357Z 2022-11-23T01:42:00.0424429Z Running tests... 2022-11-23T01:42:00.0424625Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0424849Z test_broadcast_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6301 2022-11-23T01:42:00.0425004Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6302 2022-11-23T01:42:00.0425158Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6303 2022-11-23T01:42:00.0425309Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6304 2022-11-23T01:42:00.0425582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0425693Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0425976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0426112Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0426296Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjr6no3dj 2022-11-23T01:42:00.0426493Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjr6no3dj/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0426766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0426889Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0427164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0427300Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0427472Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph0ue6k_a 2022-11-23T01:42:00.0427671Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph0ue6k_a/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0427833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0427991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0428262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0428384Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0428663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0428799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0428972Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4pm01s8p 2022-11-23T01:42:00.0429199Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4pm01s8p/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0429474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0429598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0429879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0430014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0430199Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps4obrfh1 2022-11-23T01:42:00.0430422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps4obrfh1/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0430584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0430736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0430808Z ok (2.481s) 2022-11-23T01:42:00.0430813Z 2022-11-23T01:42:00.0431008Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0431083Z Ran 1 test in 2.481s 2022-11-23T01:42:00.0431088Z 2022-11-23T01:42:00.0431149Z OK 2022-11-23T01:42:00.0431154Z 2022-11-23T01:42:00.0431237Z Generating XML reports... 2022-11-23T01:42:00.0431561Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014011.xml 2022-11-23T01:42:00.0431832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0431947Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0432225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0432363Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0432548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo65rkjbs 2022-11-23T01:42:00.0432826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo65rkjbs/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0433062Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0433067Z 2022-11-23T01:42:00.0433140Z Running tests... 2022-11-23T01:42:00.0433335Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0433564Z test_broadcast_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6392 2022-11-23T01:42:00.0433709Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6393 2022-11-23T01:42:00.0433864Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6394 2022-11-23T01:42:00.0434019Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6395 2022-11-23T01:42:00.0434292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0434415Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0434699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0434833Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0435020Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpupujgt1h 2022-11-23T01:42:00.0435202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpupujgt1h/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0435367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0435641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0435797Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0436080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0436214Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0436398Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp88_ih523 2022-11-23T01:42:00.0436591Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp88_ih523/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0436864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0437002Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0437284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0437421Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0437607Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjie59c1_ 2022-11-23T01:42:00.0437800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjie59c1_/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0438070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0438194Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0438472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0438608Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0438780Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn8hwmvsj 2022-11-23T01:42:00.0438974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn8hwmvsj/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0439139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0439300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0439463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0439569Z skip: Need at least 2 CUDA devices (1.979s) 2022-11-23T01:42:00.0439574Z 2022-11-23T01:42:00.0439772Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0439849Z Ran 1 test in 1.979s 2022-11-23T01:42:00.0439853Z 2022-11-23T01:42:00.0439914Z OK (skipped=1) 2022-11-23T01:42:00.0439933Z 2022-11-23T01:42:00.0440005Z Generating XML reports... 2022-11-23T01:42:00.0440328Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014015.xml 2022-11-23T01:42:00.0440739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0440923Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0441217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0441353Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0441538Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkoqocibj 2022-11-23T01:42:00.0441735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkoqocibj/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0441954Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0441960Z 2022-11-23T01:42:00.0442034Z Running tests... 2022-11-23T01:42:00.0442227Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0442507Z test_empty_tensors (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6447 2022-11-23T01:42:00.0442665Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6448 2022-11-23T01:42:00.0442821Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6449 2022-11-23T01:42:00.0442972Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6450 2022-11-23T01:42:00.0443247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0443356Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0443672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0443809Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0443994Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpue1n61l7 2022-11-23T01:42:00.0444272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0444397Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0444679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0444813Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0444997Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbjcb06_3 2022-11-23T01:42:00.0445181Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpue1n61l7/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0445375Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbjcb06_3/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0445649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0445775Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0446055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0446190Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0446373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx6iyq8t4 2022-11-23T01:42:00.0446566Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx6iyq8t4/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0446716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0446882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0447042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0447314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0447437Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0447717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0447851Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0448039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyyufq1mc 2022-11-23T01:42:00.0448235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyyufq1mc/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0448386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0448454Z ok (2.081s) 2022-11-23T01:42:00.0448459Z 2022-11-23T01:42:00.0448653Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0448728Z Ran 1 test in 2.082s 2022-11-23T01:42:00.0448733Z 2022-11-23T01:42:00.0448843Z OK 2022-11-23T01:42:00.0448848Z 2022-11-23T01:42:00.0448934Z Generating XML reports... 2022-11-23T01:42:00.0449262Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014019.xml 2022-11-23T01:42:00.0449537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0449649Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0449930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0450097Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0450284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgqp3pq0z 2022-11-23T01:42:00.0450480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgqp3pq0z/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0450712Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0450717Z 2022-11-23T01:42:00.0450790Z Running tests... 2022-11-23T01:42:00.0450986Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0451212Z test_gather_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6514 2022-11-23T01:42:00.0451354Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6515 2022-11-23T01:42:00.0451508Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6516 2022-11-23T01:42:00.0451663Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6517 2022-11-23T01:42:00.0451935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0452060Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0452345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0452481Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0452666Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3to_z3hr 2022-11-23T01:42:00.0452846Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3to_z3hr/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0453118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0453242Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0453523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0453658Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0453845Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpta266ocu 2022-11-23T01:42:00.0454041Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpta266ocu/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0454203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0454360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0454622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0454746Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0455030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0455165Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0455385Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnvdkwh2d 2022-11-23T01:42:00.0455583Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnvdkwh2d/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0455856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0455980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0456260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0456381Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0456564Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpadds4y0i 2022-11-23T01:42:00.0456790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpadds4y0i/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0456954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0457118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0457185Z ok (2.083s) 2022-11-23T01:42:00.0457191Z 2022-11-23T01:42:00.0457384Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0457463Z Ran 1 test in 2.083s 2022-11-23T01:42:00.0457468Z 2022-11-23T01:42:00.0457514Z OK 2022-11-23T01:42:00.0457519Z 2022-11-23T01:42:00.0457604Z Generating XML reports... 2022-11-23T01:42:00.0457928Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014022.xml 2022-11-23T01:42:00.0458197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0458322Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0458601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0458737Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0458925Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn2zo33cd 2022-11-23T01:42:00.0459119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn2zo33cd/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0459335Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0459341Z 2022-11-23T01:42:00.0459413Z Running tests... 2022-11-23T01:42:00.0459606Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0459834Z test_gather_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6581 2022-11-23T01:42:00.0459990Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6582 2022-11-23T01:42:00.0460142Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6583 2022-11-23T01:42:00.0460295Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6584 2022-11-23T01:42:00.0460570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0460680Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0460962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0461095Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0461280Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqkkdru8x 2022-11-23T01:42:00.0461479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqkkdru8x/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0461747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0461899Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0462183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0462316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0462486Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph59q08pn 2022-11-23T01:42:00.0462680Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph59q08pn/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0462955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0463110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0463390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0463527Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0463711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpla4muq0w 2022-11-23T01:42:00.0463905Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpla4muq0w/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0464055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0464217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0464375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0464648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0464772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0465049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0465185Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0465369Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn0gljtvk 2022-11-23T01:42:00.0465564Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn0gljtvk/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0465713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0465817Z skip: Need at least 2 CUDA devices (1.981s) 2022-11-23T01:42:00.0465822Z 2022-11-23T01:42:00.0466020Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0466097Z Ran 1 test in 1.981s 2022-11-23T01:42:00.0466102Z 2022-11-23T01:42:00.0466174Z OK (skipped=1) 2022-11-23T01:42:00.0466179Z 2022-11-23T01:42:00.0466262Z Generating XML reports... 2022-11-23T01:42:00.0466588Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014026.xml 2022-11-23T01:42:00.0466860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0466971Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0467251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0467384Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0467569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm9zmw_l9 2022-11-23T01:42:00.0467763Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm9zmw_l9/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0467994Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0467999Z 2022-11-23T01:42:00.0468072Z Running tests... 2022-11-23T01:42:00.0468299Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0468524Z test_gather_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6636 2022-11-23T01:42:00.0468667Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6637 2022-11-23T01:42:00.0468821Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6638 2022-11-23T01:42:00.0468972Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6639 2022-11-23T01:42:00.0469242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0469395Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0469677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0469813Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0469998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7vluahjl 2022-11-23T01:42:00.0470182Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7vluahjl/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0470345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0470615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0470738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0471019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0471158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0471342Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxs57l07x 2022-11-23T01:42:00.0471536Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxs57l07x/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0471699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0471959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0472082Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0472360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0472496Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0472762Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1v93x_y9 2022-11-23T01:42:00.0472959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1v93x_y9/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0473234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0473357Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0473636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0473756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0473943Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe9b_mvbw 2022-11-23T01:42:00.0474137Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe9b_mvbw/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0474300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0474467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0474535Z ok (2.081s) 2022-11-23T01:42:00.0474540Z 2022-11-23T01:42:00.0474770Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0474848Z Ran 1 test in 2.081s 2022-11-23T01:42:00.0474853Z 2022-11-23T01:42:00.0474901Z OK 2022-11-23T01:42:00.0474906Z 2022-11-23T01:42:00.0474993Z Generating XML reports... 2022-11-23T01:42:00.0475323Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014030.xml 2022-11-23T01:42:00.0475595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0475718Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0475998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0476197Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0476380Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7r1msu81 2022-11-23T01:42:00.0476563Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7r1msu81/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0476795Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0476800Z 2022-11-23T01:42:00.0476874Z Running tests... 2022-11-23T01:42:00.0477067Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0477302Z test_gather_noncontiguous_input (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6703 2022-11-23T01:42:00.0477459Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6704 2022-11-23T01:42:00.0477615Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6705 2022-11-23T01:42:00.0477770Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6706 2022-11-23T01:42:00.0478042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0478155Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0478438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0478575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0478761Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx50udorh 2022-11-23T01:42:00.0478959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx50udorh/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0479122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0479396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0479520Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0479800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0479922Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0480105Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn97q54s8 2022-11-23T01:42:00.0480298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn97q54s8/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0480461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0480931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0481061Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0481345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0481531Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0481704Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcl9k1el7 2022-11-23T01:42:00.0481898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcl9k1el7/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0482060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0482337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0482463Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0482742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0482918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0483104Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp34xgl43q 2022-11-23T01:42:00.0483299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp34xgl43q/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0483449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0483517Z ok (1.983s) 2022-11-23T01:42:00.0483522Z 2022-11-23T01:42:00.0483719Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0483794Z Ran 1 test in 1.983s 2022-11-23T01:42:00.0483799Z 2022-11-23T01:42:00.0483862Z OK 2022-11-23T01:42:00.0483867Z 2022-11-23T01:42:00.0483951Z Generating XML reports... 2022-11-23T01:42:00.0484274Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014033.xml 2022-11-23T01:42:00.0484546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0484654Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0484939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0485075Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0485258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2y53mjw0 2022-11-23T01:42:00.0485457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2y53mjw0/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0485685Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0485690Z 2022-11-23T01:42:00.0485762Z Running tests... 2022-11-23T01:42:00.0485957Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0486182Z test_gather_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6770 2022-11-23T01:42:00.0486324Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6771 2022-11-23T01:42:00.0486480Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6772 2022-11-23T01:42:00.0486632Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6773 2022-11-23T01:42:00.0486904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0487027Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0487307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0487442Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0487630Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ylaou2c 2022-11-23T01:42:00.0487814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ylaou2c/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0488006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0488281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0488406Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0488687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0488822Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0489015Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmploxzohok 2022-11-23T01:42:00.0489239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmploxzohok/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0489391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0489666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0489788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0490065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0490198Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0490381Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg7am9a9l 2022-11-23T01:42:00.0490575Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg7am9a9l/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0490843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0490954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0491231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0491366Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0491548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2v19_a1x 2022-11-23T01:42:00.0491738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2v19_a1x/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0491900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0492063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0492129Z ok (3.886s) 2022-11-23T01:42:00.0492133Z 2022-11-23T01:42:00.0492331Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0492394Z Ran 1 test in 3.886s 2022-11-23T01:42:00.0492400Z 2022-11-23T01:42:00.0492459Z OK 2022-11-23T01:42:00.0492463Z 2022-11-23T01:42:00.0492546Z Generating XML reports... 2022-11-23T01:42:00.0492871Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014037.xml 2022-11-23T01:42:00.0493140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0493262Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0493538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0493673Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0493843Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2n0kt2vg 2022-11-23T01:42:00.0494036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2n0kt2vg/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0494262Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0494267Z 2022-11-23T01:42:00.0494338Z Running tests... 2022-11-23T01:42:00.0494577Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0494807Z test_gather_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6861 2022-11-23T01:42:00.0494960Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6862 2022-11-23T01:42:00.0495112Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6863 2022-11-23T01:42:00.0495263Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6864 2022-11-23T01:42:00.0495523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0495673Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0495953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0496089Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0496272Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8pkrtmrd 2022-11-23T01:42:00.0496467Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8pkrtmrd/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0496627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0496896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0497006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0497284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0497419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0497602Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp06c0iwyk 2022-11-23T01:42:00.0497796Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp06c0iwyk/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0498065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0498189Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0498465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0498600Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0498770Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp5wst_ii 2022-11-23T01:42:00.0498964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp5wst_ii/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0499125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0499398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0499519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0499794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0499931Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0500113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzvilk_4p 2022-11-23T01:42:00.0500293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzvilk_4p/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0500457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0500618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0500721Z skip: Need at least 2 CUDA devices (1.990s) 2022-11-23T01:42:00.0500756Z 2022-11-23T01:42:00.0500951Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0501025Z Ran 1 test in 1.990s 2022-11-23T01:42:00.0501031Z 2022-11-23T01:42:00.0501103Z OK (skipped=1) 2022-11-23T01:42:00.0501107Z 2022-11-23T01:42:00.0501191Z Generating XML reports... 2022-11-23T01:42:00.0501513Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014042.xml 2022-11-23T01:42:00.0501774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0501897Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0502205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0502343Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0502530Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvd5vags0 2022-11-23T01:42:00.0502726Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvd5vags0/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0502952Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0502957Z 2022-11-23T01:42:00.0503028Z Running tests... 2022-11-23T01:42:00.0503223Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0503442Z test_multi_device_constructor (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6916 2022-11-23T01:42:00.0503597Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6917 2022-11-23T01:42:00.0503749Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6918 2022-11-23T01:42:00.0503896Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6919 2022-11-23T01:42:00.0504167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0504290Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0504567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0504701Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0504871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpus04fki2 2022-11-23T01:42:00.0505141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0505265Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0505546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0505681Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0505865Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpudulgbz9 2022-11-23T01:42:00.0506057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpus04fki2/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0506247Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpudulgbz9/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0506408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0506557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0506830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0506953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0507268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0507404Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0507585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1b14lm19 2022-11-23T01:42:00.0507780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1b14lm19/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0507941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0508198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0508319Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0508626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0508763Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0508949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq8_7fuh6 2022-11-23T01:42:00.0509139Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq8_7fuh6/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0509300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0509367Z ok (2.181s) 2022-11-23T01:42:00.0509372Z 2022-11-23T01:42:00.0509566Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0509628Z Ran 1 test in 2.181s 2022-11-23T01:42:00.0509633Z 2022-11-23T01:42:00.0509692Z OK 2022-11-23T01:42:00.0509697Z 2022-11-23T01:42:00.0509782Z Generating XML reports... 2022-11-23T01:42:00.0510107Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014046.xml 2022-11-23T01:42:00.0510378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0510502Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0510777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0510912Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0511082Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplpyg2jil 2022-11-23T01:42:00.0511279Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplpyg2jil/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0511505Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0511512Z 2022-11-23T01:42:00.0511583Z Running tests... 2022-11-23T01:42:00.0511776Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0511995Z test_reduce_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6987 2022-11-23T01:42:00.0512150Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6988 2022-11-23T01:42:00.0512304Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6989 2022-11-23T01:42:00.0512454Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6990 2022-11-23T01:42:00.0512782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0512905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0513187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0513323Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0513506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkl4zmql_ 2022-11-23T01:42:00.0513738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkl4zmql_/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0513901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0514175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0514286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0514564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0514698Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0514883Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl9a_2pif 2022-11-23T01:42:00.0515179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0515301Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0515578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0515710Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0515893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppszky0jw 2022-11-23T01:42:00.0516076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl9a_2pif/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0516265Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppszky0jw/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0516427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0516591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0516863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0516986Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0517265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0517396Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0517566Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzn71xs3h 2022-11-23T01:42:00.0517759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzn71xs3h/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0517918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0517987Z ok (2.082s) 2022-11-23T01:42:00.0517992Z 2022-11-23T01:42:00.0518186Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0518261Z Ran 1 test in 2.082s 2022-11-23T01:42:00.0518266Z 2022-11-23T01:42:00.0518326Z OK 2022-11-23T01:42:00.0518331Z 2022-11-23T01:42:00.0518415Z Generating XML reports... 2022-11-23T01:42:00.0518737Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014050.xml 2022-11-23T01:42:00.0518995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0519116Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0519394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0519528Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0519715Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp99zqxqzr 2022-11-23T01:42:00.0519908Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp99zqxqzr/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0520135Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0520169Z 2022-11-23T01:42:00.0520242Z Running tests... 2022-11-23T01:42:00.0520424Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0520805Z test_reduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7054 2022-11-23T01:42:00.0521002Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7055 2022-11-23T01:42:00.0521155Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7056 2022-11-23T01:42:00.0521305Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7057 2022-11-23T01:42:00.0521638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0521760Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0522040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0522175Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0522347Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkac__vpa 2022-11-23T01:42:00.0522542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkac__vpa/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0522704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0522975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0523099Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0523379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0523512Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0523695Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvn0gvn9u 2022-11-23T01:42:00.0523878Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvn0gvn9u/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0524146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0524268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0524546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0524681Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0524864Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0591b7ka 2022-11-23T01:42:00.0525056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0591b7ka/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0525220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0525385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0525646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0525767Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0526046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0526180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0526365Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4s0hxumj 2022-11-23T01:42:00.0526557Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4s0hxumj/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0526758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0526864Z skip: Need at least 2 CUDA devices (1.990s) 2022-11-23T01:42:00.0526870Z 2022-11-23T01:42:00.0527065Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0527128Z Ran 1 test in 1.990s 2022-11-23T01:42:00.0527133Z 2022-11-23T01:42:00.0527203Z OK (skipped=1) 2022-11-23T01:42:00.0527207Z 2022-11-23T01:42:00.0527290Z Generating XML reports... 2022-11-23T01:42:00.0527615Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014053.xml 2022-11-23T01:42:00.0527886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0528038Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0528317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0528453Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0528624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpop7gxlhc 2022-11-23T01:42:00.0528819Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpop7gxlhc/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0529049Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0529055Z 2022-11-23T01:42:00.0529126Z Running tests... 2022-11-23T01:42:00.0529320Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0529541Z test_reduce_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7109 2022-11-23T01:42:00.0529698Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7110 2022-11-23T01:42:00.0529851Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7111 2022-11-23T01:42:00.0530004Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7112 2022-11-23T01:42:00.0530266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0530388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0530667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0530802Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0530987Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc8goiper 2022-11-23T01:42:00.0531183Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc8goiper/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0531345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0531621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0531732Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0532010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0532196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0532448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpygi04q9b 2022-11-23T01:42:00.0532667Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpygi04q9b/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0532985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0533122Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0533563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0533770Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0533946Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwh2kgfp7 2022-11-23T01:42:00.0534198Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwh2kgfp7/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0534362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0534526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0534808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0534965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0535248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0535383Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0535556Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn0y10a5b 2022-11-23T01:42:00.0535750Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn0y10a5b/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0535914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0535981Z ok (2.181s) 2022-11-23T01:42:00.0535988Z 2022-11-23T01:42:00.0536185Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0536259Z Ran 1 test in 2.182s 2022-11-23T01:42:00.0536264Z 2022-11-23T01:42:00.0536327Z OK 2022-11-23T01:42:00.0536332Z 2022-11-23T01:42:00.0536414Z Generating XML reports... 2022-11-23T01:42:00.0536737Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014057.xml 2022-11-23T01:42:00.0537000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0537122Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0537402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0537538Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0537722Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyg5hlqjl 2022-11-23T01:42:00.0537917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyg5hlqjl/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0538146Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0538151Z 2022-11-23T01:42:00.0538223Z Running tests... 2022-11-23T01:42:00.0538405Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0538629Z test_reduce_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7176 2022-11-23T01:42:00.0538785Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7177 2022-11-23T01:42:00.0538936Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7178 2022-11-23T01:42:00.0539086Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7179 2022-11-23T01:42:00.0539358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0539480Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0539765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0539900Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0540072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgcfp5dfn 2022-11-23T01:42:00.0540312Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgcfp5dfn/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0540475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0540747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0540869Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0541149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0541284Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0541495Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmploa64o6b 2022-11-23T01:42:00.0541754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0541879Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0542161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0542297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0542481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy8ja1d83 2022-11-23T01:42:00.0542674Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmploa64o6b/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0542862Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy8ja1d83/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0543029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0543190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0543452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0543574Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0543854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0543988Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0544171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7le2t284 2022-11-23T01:42:00.0544363Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7le2t284/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0544526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0544594Z ok (3.592s) 2022-11-23T01:42:00.0544601Z 2022-11-23T01:42:00.0544799Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0544861Z Ran 1 test in 3.593s 2022-11-23T01:42:00.0544866Z 2022-11-23T01:42:00.0544928Z OK 2022-11-23T01:42:00.0544934Z 2022-11-23T01:42:00.0545017Z Generating XML reports... 2022-11-23T01:42:00.0545341Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014101.xml 2022-11-23T01:42:00.0545613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0545735Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0546014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0546151Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0546322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfwo6ki1m 2022-11-23T01:42:00.0546515Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfwo6ki1m/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0546771Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0546777Z 2022-11-23T01:42:00.0546850Z Running tests... 2022-11-23T01:42:00.0547046Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0547274Z test_reduce_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7267 2022-11-23T01:42:00.0547427Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7268 2022-11-23T01:42:00.0547582Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7269 2022-11-23T01:42:00.0547762Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7270 2022-11-23T01:42:00.0548020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0548140Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0548425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0548559Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0548743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwy68a6h1 2022-11-23T01:42:00.0548938Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwy68a6h1/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0549210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0549332Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0549599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0549733Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0549917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp04i44s_j 2022-11-23T01:42:00.0550111Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp04i44s_j/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0550384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0550506Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0550784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0550918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0551102Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6653t4eh 2022-11-23T01:42:00.0551280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6653t4eh/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0551443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0551606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0551767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0552037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0552159Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0552440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0552576Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0552831Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptnyilg4_ 2022-11-23T01:42:00.0553027Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptnyilg4_/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0553222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0553328Z skip: Need at least 2 CUDA devices (1.988s) 2022-11-23T01:42:00.0553333Z 2022-11-23T01:42:00.0553532Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0553607Z Ran 1 test in 1.988s 2022-11-23T01:42:00.0553612Z 2022-11-23T01:42:00.0553681Z OK (skipped=1) 2022-11-23T01:42:00.0553686Z 2022-11-23T01:42:00.0553769Z Generating XML reports... 2022-11-23T01:42:00.0554092Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014106.xml 2022-11-23T01:42:00.0554384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0554510Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0554789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0554927Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0555112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplau9s9pq 2022-11-23T01:42:00.0555307Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplau9s9pq/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0555536Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0555541Z 2022-11-23T01:42:00.0555611Z Running tests... 2022-11-23T01:42:00.0555794Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0556012Z test_round_robin (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7322 2022-11-23T01:42:00.0556166Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7323 2022-11-23T01:42:00.0556321Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7324 2022-11-23T01:42:00.0556470Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7325 2022-11-23T01:42:00.0556745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0556867Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0557148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0557282Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0557452Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcv_zzd77 2022-11-23T01:42:00.0557647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcv_zzd77/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0557808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0558080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0558202Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0558480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0558612Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0558794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppg673a68 2022-11-23T01:42:00.0558975Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppg673a68/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0559247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0559369Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0559679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0559815Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0559998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbloyx_tf 2022-11-23T01:42:00.0560190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbloyx_tf/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0560459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0560702Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0561102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0561236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0561421Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsfaqunm1 2022-11-23T01:42:00.0561618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsfaqunm1/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0561780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0561941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0562102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0562535Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T01:42:00.0562954Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T01:42:00.0563363Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T01:42:00.0563769Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T01:42:00.0563839Z ok (2.079s) 2022-11-23T01:42:00.0563844Z 2022-11-23T01:42:00.0564029Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0564106Z Ran 1 test in 2.079s 2022-11-23T01:42:00.0564113Z 2022-11-23T01:42:00.0564173Z OK 2022-11-23T01:42:00.0564178Z 2022-11-23T01:42:00.0564262Z Generating XML reports... 2022-11-23T01:42:00.0564585Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014110.xml 2022-11-23T01:42:00.0564858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0564980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0565259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0565396Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0565569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpng92o6y3 2022-11-23T01:42:00.0565803Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpng92o6y3/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0566033Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0566038Z 2022-11-23T01:42:00.0566111Z Running tests... 2022-11-23T01:42:00.0566306Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0566538Z test_round_robin_create_destroy (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7401 2022-11-23T01:42:00.0566693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7402 2022-11-23T01:42:00.0566846Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7403 2022-11-23T01:42:00.0567015Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7404 2022-11-23T01:42:00.0567287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0567414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0567692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0567826Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0568010Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptlhz7cge 2022-11-23T01:42:00.0568206Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptlhz7cge/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0568367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0568642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0568751Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0569030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0569164Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0569345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8_uu9n37 2022-11-23T01:42:00.0569614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0569734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0570015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0570149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0570322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfknqgxta 2022-11-23T01:42:00.0570513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8_uu9n37/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0570703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfknqgxta/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0570974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0571096Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0571373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0571507Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0571690Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpahpzsx5y 2022-11-23T01:42:00.0571888Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpahpzsx5y/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0572039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0572230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0572393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0572809Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T01:42:00.0573220Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T01:42:00.0573662Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T01:42:00.0574063Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T01:42:00.0574465Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T01:42:00.0574865Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T01:42:00.0575264Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T01:42:00.0575658Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T01:42:00.0575732Z ok (2.290s) 2022-11-23T01:42:00.0575737Z 2022-11-23T01:42:00.0575934Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0576008Z Ran 1 test in 2.290s 2022-11-23T01:42:00.0576013Z 2022-11-23T01:42:00.0576073Z OK 2022-11-23T01:42:00.0576080Z 2022-11-23T01:42:00.0576150Z Generating XML reports... 2022-11-23T01:42:00.0576476Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014113.xml 2022-11-23T01:42:00.0576749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0576871Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0577152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0577286Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0577474Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpptvcbxtc 2022-11-23T01:42:00.0577672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpptvcbxtc/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0577970Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0577976Z 2022-11-23T01:42:00.0578035Z Running tests... 2022-11-23T01:42:00.0578230Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0578452Z test_scatter_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7504 2022-11-23T01:42:00.0578607Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7505 2022-11-23T01:42:00.0578758Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7506 2022-11-23T01:42:00.0578908Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7507 2022-11-23T01:42:00.0579212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0579334Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0579605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0579740Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0579926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp49hpl0hr 2022-11-23T01:42:00.0580124Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp49hpl0hr/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0580287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0580557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0580683Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0580963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0581098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0581270Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl35drisz 2022-11-23T01:42:00.0581463Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl35drisz/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0581626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0581899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0582021Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0582300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0582433Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0582616Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps6is_44s 2022-11-23T01:42:00.0582798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps6is_44s/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0582960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0583231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0583352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0583632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0583766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0583952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbgk0kq_k 2022-11-23T01:42:00.0584149Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbgk0kq_k/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0584357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0584413Z ok (2.079s) 2022-11-23T01:42:00.0584418Z 2022-11-23T01:42:00.0584616Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0584690Z Ran 1 test in 2.079s 2022-11-23T01:42:00.0584695Z 2022-11-23T01:42:00.0584754Z OK 2022-11-23T01:42:00.0584758Z 2022-11-23T01:42:00.0584842Z Generating XML reports... 2022-11-23T01:42:00.0585163Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014117.xml 2022-11-23T01:42:00.0585435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0585590Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0585856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0585992Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0586175Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprfa4bjpw 2022-11-23T01:42:00.0586369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprfa4bjpw/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0586597Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0586601Z 2022-11-23T01:42:00.0586674Z Running tests... 2022-11-23T01:42:00.0586867Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0587091Z test_scatter_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7571 2022-11-23T01:42:00.0587247Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7572 2022-11-23T01:42:00.0587386Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7573 2022-11-23T01:42:00.0587539Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7574 2022-11-23T01:42:00.0587808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0587931Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0588212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0588346Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0588529Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwam8_8zx 2022-11-23T01:42:00.0588724Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwam8_8zx/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0588875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0589153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0589276Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0589555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0589688Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0589870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgm05cdbq 2022-11-23T01:42:00.0590064Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgm05cdbq/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0590229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0590500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0590610Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0590921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0591054Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0591236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2wu2fgpu 2022-11-23T01:42:00.0591430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2wu2fgpu/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0591699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0591820Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0592125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0592257Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0592431Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmsaslf9t 2022-11-23T01:42:00.0592698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmsaslf9t/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0592862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0593024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0593127Z skip: Need at least 2 CUDA devices (2.080s) 2022-11-23T01:42:00.0593132Z 2022-11-23T01:42:00.0593331Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0593409Z Ran 1 test in 2.080s 2022-11-23T01:42:00.0593414Z 2022-11-23T01:42:00.0593484Z OK (skipped=1) 2022-11-23T01:42:00.0593489Z 2022-11-23T01:42:00.0593559Z Generating XML reports... 2022-11-23T01:42:00.0593883Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014121.xml 2022-11-23T01:42:00.0594157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0594279Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0594561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0594695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0594880Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgeczn3s4 2022-11-23T01:42:00.0595075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgeczn3s4/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0595303Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0595308Z 2022-11-23T01:42:00.0595367Z Running tests... 2022-11-23T01:42:00.0595561Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0595786Z test_scatter_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7626 2022-11-23T01:42:00.0595941Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7627 2022-11-23T01:42:00.0596092Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7628 2022-11-23T01:42:00.0596242Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7629 2022-11-23T01:42:00.0596513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0596637Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0596906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0597041Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0597261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl90iwl2u 2022-11-23T01:42:00.0597456Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl90iwl2u/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0597620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0597889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0598011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0598287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0598452Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0598624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphp1wtx78 2022-11-23T01:42:00.0598819Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphp1wtx78/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0598981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0599252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0599374Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0599655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0599788Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0599970Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpupore0dy 2022-11-23T01:42:00.0600155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpupore0dy/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0600431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0600551Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0601044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0601178Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0601361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqtiybip2 2022-11-23T01:42:00.0601555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqtiybip2/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0601717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0601883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0601937Z ok (2.182s) 2022-11-23T01:42:00.0601942Z 2022-11-23T01:42:00.0602138Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0602214Z Ran 1 test in 2.183s 2022-11-23T01:42:00.0602219Z 2022-11-23T01:42:00.0602280Z OK 2022-11-23T01:42:00.0602285Z 2022-11-23T01:42:00.0602367Z Generating XML reports... 2022-11-23T01:42:00.0602693Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014125.xml 2022-11-23T01:42:00.0602964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0603086Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0603351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0603488Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0603668Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsvd12ht5 2022-11-23T01:42:00.0603921Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsvd12ht5/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0604153Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0604158Z 2022-11-23T01:42:00.0604229Z Running tests... 2022-11-23T01:42:00.0604424Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0604647Z test_scatter_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7693 2022-11-23T01:42:00.0604801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7694 2022-11-23T01:42:00.0604941Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7695 2022-11-23T01:42:00.0605132Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7696 2022-11-23T01:42:00.0605412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0605537Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0605815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0605950Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0606134Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl6ld8y84 2022-11-23T01:42:00.0606329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl6ld8y84/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0606478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0606753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0606874Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0607156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0607290Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0607470Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6k8sty55 2022-11-23T01:42:00.0607661Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6k8sty55/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0607824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0608099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0608210Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0608492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0608625Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0608811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjrw5l_xf 2022-11-23T01:42:00.0609005Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjrw5l_xf/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0609168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0609438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0609560Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0609840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0609963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0610147Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj4gb7nls 2022-11-23T01:42:00.0610371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj4gb7nls/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0610534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0610601Z ok (3.785s) 2022-11-23T01:42:00.0610606Z 2022-11-23T01:42:00.0610800Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0610874Z Ran 1 test in 3.786s 2022-11-23T01:42:00.0610879Z 2022-11-23T01:42:00.0610938Z OK 2022-11-23T01:42:00.0610942Z 2022-11-23T01:42:00.0611013Z Generating XML reports... 2022-11-23T01:42:00.0611342Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014129.xml 2022-11-23T01:42:00.0611655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0611781Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0612061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0612194Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0612377Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy9uh2brw 2022-11-23T01:42:00.0612571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy9uh2brw/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0612798Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0612803Z 2022-11-23T01:42:00.0612863Z Running tests... 2022-11-23T01:42:00.0613056Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0613290Z test_scatter_stress_cuda (__main__.ProcessGroupGlooTest) ... skip: Test is flaky, see https://github.com/pytorch/pytorch/issues/15963 (0.000s) 2022-11-23T01:42:00.0613296Z 2022-11-23T01:42:00.0613488Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0613564Z Ran 1 test in 0.001s 2022-11-23T01:42:00.0613569Z 2022-11-23T01:42:00.0613639Z OK (skipped=1) 2022-11-23T01:42:00.0613643Z 2022-11-23T01:42:00.0613726Z Generating XML reports... 2022-11-23T01:42:00.0614050Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014134.xml 2022-11-23T01:42:00.0614323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0614434Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0614713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0614849Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0615032Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu_fdg9bi 2022-11-23T01:42:00.0615226Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu_fdg9bi/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0615453Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0615458Z 2022-11-23T01:42:00.0615529Z Running tests... 2022-11-23T01:42:00.0615723Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0615935Z test_send_recv_all_to_all (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7794 2022-11-23T01:42:00.0616089Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7795 2022-11-23T01:42:00.0616240Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7796 2022-11-23T01:42:00.0616391Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7797 2022-11-23T01:42:00.0616662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0616815Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0617096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0617230Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0617413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpor8ucj14 2022-11-23T01:42:00.0617594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpor8ucj14/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0617755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0618055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0618177Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0618456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0618589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0618772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppwdfys2s 2022-11-23T01:42:00.0618967Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppwdfys2s/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0619115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0619383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0619508Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0619789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0619921Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0620106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxd6m48_n 2022-11-23T01:42:00.0620301Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxd6m48_n/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0620568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0620689Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0620955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0621087Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0621271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo6nfu481 2022-11-23T01:42:00.0621463Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo6nfu481/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0621626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0621786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0621852Z ok (1.982s) 2022-11-23T01:42:00.0621857Z 2022-11-23T01:42:00.0622050Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0622112Z Ran 1 test in 1.982s 2022-11-23T01:42:00.0622116Z 2022-11-23T01:42:00.0622175Z OK 2022-11-23T01:42:00.0622179Z 2022-11-23T01:42:00.0622262Z Generating XML reports... 2022-11-23T01:42:00.0622582Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014135.xml 2022-11-23T01:42:00.0622853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0622975Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0623291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0623426Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0623609Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl04o86ep 2022-11-23T01:42:00.0623790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl04o86ep/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0624019Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0624024Z 2022-11-23T01:42:00.0624095Z Running tests... 2022-11-23T01:42:00.0624286Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0624513Z test_sparse_allreduce_basics (__main__.ProcessGroupGlooTest) ... skip: intermittent failures on Windows, in CI (0.000s) 2022-11-23T01:42:00.0624518Z 2022-11-23T01:42:00.0624709Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0624787Z Ran 1 test in 0.000s 2022-11-23T01:42:00.0624792Z 2022-11-23T01:42:00.0624861Z OK (skipped=1) 2022-11-23T01:42:00.0624865Z 2022-11-23T01:42:00.0624935Z Generating XML reports... 2022-11-23T01:42:00.0625260Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014139.xml 2022-11-23T01:42:00.0625531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0625654Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0625934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0626070Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0626254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8kn8rh6a 2022-11-23T01:42:00.0626450Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8kn8rh6a/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0626678Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0626683Z 2022-11-23T01:42:00.0626743Z Running tests... 2022-11-23T01:42:00.0626937Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0627175Z test_sparse_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7871 2022-11-23T01:42:00.0627329Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7872 2022-11-23T01:42:00.0627480Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7873 2022-11-23T01:42:00.0627634Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7874 2022-11-23T01:42:00.0627906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0628032Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0628313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0628434Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0628616Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpklp7rcmd 2022-11-23T01:42:00.0628811Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpklp7rcmd/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0628973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0629245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0629367Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0629693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0629830Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0629999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8osgsma3 2022-11-23T01:42:00.0630191Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8osgsma3/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0630354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0630624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0630777Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0631055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0631189Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0631373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjx8lw_ze 2022-11-23T01:42:00.0631566Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjx8lw_ze/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0631715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0631985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0632106Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0632385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0632521Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0632782Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps03prwmb 2022-11-23T01:42:00.0632981Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps03prwmb/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0633143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0633234Z skip: Need at least 2 CUDA devices (1.978s) 2022-11-23T01:42:00.0633251Z 2022-11-23T01:42:00.0633438Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0633512Z Ran 1 test in 1.978s 2022-11-23T01:42:00.0633517Z 2022-11-23T01:42:00.0633588Z OK (skipped=1) 2022-11-23T01:42:00.0633592Z 2022-11-23T01:42:00.0633675Z Generating XML reports... 2022-11-23T01:42:00.0634002Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014141.xml 2022-11-23T01:42:00.0634275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0634396Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0634681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0634801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0634986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl09yj2op 2022-11-23T01:42:00.0635180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl09yj2op/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0635409Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0635415Z 2022-11-23T01:42:00.0635486Z Running tests... 2022-11-23T01:42:00.0635683Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0635915Z test_sparse_allreduce_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7926 2022-11-23T01:42:00.0636107Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7927 2022-11-23T01:42:00.0636260Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7928 2022-11-23T01:42:00.0636398Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7929 2022-11-23T01:42:00.0636672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0636795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0637075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0637210Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0637425Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkz3wkmrp 2022-11-23T01:42:00.0637621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkz3wkmrp/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0637891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0638000Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0638281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0638414Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0638602Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy0ye7j_y 2022-11-23T01:42:00.0638797Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy0ye7j_y/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0638962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:42:00.0639123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T01:42:00.0639394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0639515Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0639779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0639912Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0640095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf0lke6yn 2022-11-23T01:42:00.0640289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf0lke6yn/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0640452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:42:00.0640917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0641043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0641326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0641447Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0641633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz3o6c5r4 2022-11-23T01:42:00.0641827Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz3o6c5r4/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0641988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T01:42:00.0642058Z ok (2.081s) 2022-11-23T01:42:00.0642062Z 2022-11-23T01:42:00.0642261Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0642336Z Ran 1 test in 2.081s 2022-11-23T01:42:00.0642341Z 2022-11-23T01:42:00.0642401Z OK 2022-11-23T01:42:00.0642406Z 2022-11-23T01:42:00.0642488Z Generating XML reports... 2022-11-23T01:42:00.0642858Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014144.xml 2022-11-23T01:42:00.0643133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0643257Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0643535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0643669Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0643853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzn5lvmru 2022-11-23T01:42:00.0644084Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzn5lvmru/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0644313Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0644318Z 2022-11-23T01:42:00.0644377Z Running tests... 2022-11-23T01:42:00.0644574Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0644696Z test_forward_backward (__main__.ReducerTest) ... ok (0.004s) 2022-11-23T01:42:00.0644700Z 2022-11-23T01:42:00.0644893Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0644967Z Ran 1 test in 0.011s 2022-11-23T01:42:00.0644972Z 2022-11-23T01:42:00.0645031Z OK 2022-11-23T01:42:00.0645036Z 2022-11-23T01:42:00.0645118Z Generating XML reports... 2022-11-23T01:42:00.0645416Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123014148.xml 2022-11-23T01:42:00.0645690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0645800Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0646083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0646217Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0646400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyz6l5kfs 2022-11-23T01:42:00.0646593Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyz6l5kfs/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0646819Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0646824Z 2022-11-23T01:42:00.0646895Z Running tests... 2022-11-23T01:42:00.0647086Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0647741Z test_forward_backward_optimizer (__main__.ReducerTest) ... [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T01:42:00.0647811Z ok (0.007s) 2022-11-23T01:42:00.0647815Z 2022-11-23T01:42:00.0647997Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0648070Z Ran 1 test in 0.011s 2022-11-23T01:42:00.0648075Z 2022-11-23T01:42:00.0648135Z OK 2022-11-23T01:42:00.0648140Z 2022-11-23T01:42:00.0648223Z Generating XML reports... 2022-11-23T01:42:00.0648518Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123014149.xml 2022-11-23T01:42:00.0648797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0648919Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0649233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0649370Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0649543Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbxv_cn0b 2022-11-23T01:42:00.0649738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbxv_cn0b/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0649966Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0649970Z 2022-11-23T01:42:00.0650042Z Running tests... 2022-11-23T01:42:00.0650266Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0650412Z test_forward_backward_unused_parameters (__main__.ReducerTest) ... ok (0.004s) 2022-11-23T01:42:00.0650417Z 2022-11-23T01:42:00.0650606Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0650682Z Ran 1 test in 0.011s 2022-11-23T01:42:00.0650687Z 2022-11-23T01:42:00.0650733Z OK 2022-11-23T01:42:00.0650738Z 2022-11-23T01:42:00.0650819Z Generating XML reports... 2022-11-23T01:42:00.0651114Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123014151.xml 2022-11-23T01:42:00.0651386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0651508Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0651788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0651925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0652105Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_xv6b0sc 2022-11-23T01:42:00.0652303Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_xv6b0sc/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0652519Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0652524Z 2022-11-23T01:42:00.0652596Z Running tests... 2022-11-23T01:42:00.0652789Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0652918Z test_multi_dtype_multi_bucket (__main__.ReducerTest) ... ok (0.002s) 2022-11-23T01:42:00.0652923Z 2022-11-23T01:42:00.0653114Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0653188Z Ran 1 test in 0.011s 2022-11-23T01:42:00.0653193Z 2022-11-23T01:42:00.0653253Z OK 2022-11-23T01:42:00.0653258Z 2022-11-23T01:42:00.0653339Z Generating XML reports... 2022-11-23T01:42:00.0653619Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123014152.xml 2022-11-23T01:42:00.0653893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0654017Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0654301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0654434Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0654619Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzt2cw9mz 2022-11-23T01:42:00.0654814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzt2cw9mz/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0655042Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0655049Z 2022-11-23T01:42:00.0655120Z Running tests... 2022-11-23T01:42:00.0655303Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0655433Z test_multi_dtype_single_bucket (__main__.ReducerTest) ... ok (0.005s) 2022-11-23T01:42:00.0655468Z 2022-11-23T01:42:00.0655662Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0655735Z Ran 1 test in 0.011s 2022-11-23T01:42:00.0655740Z 2022-11-23T01:42:00.0655800Z OK 2022-11-23T01:42:00.0655805Z 2022-11-23T01:42:00.0655887Z Generating XML reports... 2022-11-23T01:42:00.0656180Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123014154.xml 2022-11-23T01:42:00.0656451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0656562Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0656873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0657008Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0657194Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3iex9vs4 2022-11-23T01:42:00.0657388Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3iex9vs4/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0657617Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0657621Z 2022-11-23T01:42:00.0657693Z Running tests... 2022-11-23T01:42:00.0657886Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0658017Z test_single_dtype_single_bucket (__main__.ReducerTest) ... ok (0.002s) 2022-11-23T01:42:00.0658021Z 2022-11-23T01:42:00.0658199Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0658276Z Ran 1 test in 0.011s 2022-11-23T01:42:00.0658281Z 2022-11-23T01:42:00.0658340Z OK 2022-11-23T01:42:00.0658345Z 2022-11-23T01:42:00.0658483Z Generating XML reports... 2022-11-23T01:42:00.0658810Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123014155.xml 2022-11-23T01:42:00.0659129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0665664Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0666031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0666174Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0666351Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyn5ya5aa 2022-11-23T01:42:00.0666556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyn5ya5aa/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0666800Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0666808Z 2022-11-23T01:42:00.0666882Z Running tests... 2022-11-23T01:42:00.0667086Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0667321Z test_logging_init (__main__.RendezvousEnvTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:42:00.0667625Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T01:42:00.0667694Z ok (0.268s) 2022-11-23T01:42:00.0667700Z 2022-11-23T01:42:00.0667897Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0667961Z Ran 1 test in 0.268s 2022-11-23T01:42:00.0667979Z 2022-11-23T01:42:00.0668027Z OK 2022-11-23T01:42:00.0668034Z 2022-11-23T01:42:00.0668118Z Generating XML reports... 2022-11-23T01:42:00.0668434Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-RendezvousEnvTest-20221123014157.xml 2022-11-23T01:42:00.0668713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:42:00.0668997Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:42:00.0669283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:42:00.0669423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:42:00.0669610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyn7etija 2022-11-23T01:42:00.0669795Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyn7etija/_remote_module_non_scriptable.py 2022-11-23T01:42:00.0670027Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T01:42:00.0670070Z 2022-11-23T01:42:00.0670144Z Running tests... 2022-11-23T01:42:00.0670343Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0671017Z test_default_store_timeout_gloo (__main__.TimeoutTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/74714 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.257s) 2022-11-23T01:42:00.0671023Z 2022-11-23T01:42:00.0671217Z ---------------------------------------------------------------------- 2022-11-23T01:42:00.0671292Z Ran 1 test in 0.257s 2022-11-23T01:42:00.0671297Z 2022-11-23T01:42:00.0671369Z OK (skipped=1) 2022-11-23T01:42:00.0671374Z 2022-11-23T01:42:00.0671458Z Generating XML reports... 2022-11-23T01:42:00.0671758Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-TimeoutTest-20221123014159.xml 2022-11-23T01:42:00.0671765Z 2022-11-23T01:42:00.0672111Z ##[endgroup] 2022-11-23T01:42:00.0672437Z FINISHED PRINTING LOG FILE of distributed/test_c10d_gloo (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_gloo_j6rogl31) 2022-11-23T01:42:00.0672442Z 2022-11-23T01:42:00.2969100Z 2022-11-23T01:42:00.2969508Z real 37m58.816s 2022-11-23T01:42:00.2969656Z user 83m11.713s 2022-11-23T01:42:00.2969781Z sys 20m51.845s 2022-11-23T01:42:00.2969929Z + assert_git_not_dirty 2022-11-23T01:42:00.2970365Z + [[ linux-focal-py3.7-gcc7 != *rocm* ]] 2022-11-23T01:42:00.2970606Z + [[ linux-focal-py3.7-gcc7 != *xla* ]] 2022-11-23T01:42:00.2971966Z ++ git status --porcelain 2022-11-23T01:42:11.0901088Z + git_status= 2022-11-23T01:42:11.0901476Z + [[ -n '' ]] 2022-11-23T01:42:11.0901740Z + [[ linux-focal-py3.7-gcc7 == *cuda* ]] 2022-11-23T01:42:11.0901989Z + [[ 2 == 1 ]] 2022-11-23T01:42:11.0968006Z Prepare all required actions 2022-11-23T01:42:11.0968331Z Getting action download info 2022-11-23T01:42:11.4076501Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2022-11-23T01:42:11.6406418Z ##[group]Run ./.github/actions/get-workflow-job-id 2022-11-23T01:42:11.6406636Z with: 2022-11-23T01:42:11.6406954Z github-token: *** 2022-11-23T01:42:11.6407109Z env: 2022-11-23T01:42:11.6407282Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:42:11.6407564Z DOCKER_CONTAINER_ID: 6e200e7a94c0c354a3a0a6fc7e5a55f6d42c9daee3c19a19715f86f8daa96d25 2022-11-23T01:42:11.6407812Z ##[endgroup] 2022-11-23T01:42:11.6431810Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2022-11-23T01:42:11.6432044Z with: 2022-11-23T01:42:11.6432190Z shell: bash 2022-11-23T01:42:11.6432364Z timeout_minutes: 10 2022-11-23T01:42:11.6432547Z max_attempts: 5 2022-11-23T01:42:11.6432856Z retry_wait_seconds: 30 2022-11-23T01:42:11.6433241Z command: set -eux python3 -m pip install requests==2.26.0 GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}") echo "job-id=${GHA_WORKFLOW_JOB_ID}" >> "${GITHUB_OUTPUT}" 2022-11-23T01:42:11.6433616Z polling_interval_seconds: 1 2022-11-23T01:42:11.6433811Z warning_on_retry: true 2022-11-23T01:42:11.6434004Z continue_on_error: false 2022-11-23T01:42:11.6434167Z env: 2022-11-23T01:42:11.6434338Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:42:11.6434615Z DOCKER_CONTAINER_ID: 6e200e7a94c0c354a3a0a6fc7e5a55f6d42c9daee3c19a19715f86f8daa96d25 2022-11-23T01:42:11.6434996Z GITHUB_TOKEN: *** 2022-11-23T01:42:11.6435157Z ##[endgroup] 2022-11-23T01:42:11.9698107Z + python3 -m pip install requests==2.26.0 2022-11-23T01:42:12.1895054Z Defaulting to user installation because normal site-packages is not writeable 2022-11-23T01:42:12.2117908Z Requirement already satisfied: requests==2.26.0 in /home/ec2-user/.local/lib/python3.7/site-packages (2.26.0) 2022-11-23T01:42:12.2245531Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2022.9.24) 2022-11-23T01:42:12.2253832Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (1.26.12) 2022-11-23T01:42:12.2405780Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (3.4) 2022-11-23T01:42:12.2417119Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2.0.12) 2022-11-23T01:42:12.4225176Z ++ python3 .github/scripts/get_workflow_job_id.py 3528293562 i-03a718e248ce6299b 2022-11-23T01:42:15.2833611Z + GHA_WORKFLOW_JOB_ID=9655067063 2022-11-23T01:42:15.2835038Z + echo job-id=9655067063 2022-11-23T01:42:15.9705843Z Command completed after 1 attempt(s). 2022-11-23T01:42:15.9810907Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2022-11-23T01:42:15.9811158Z kill "$MONITOR_SCRIPT_PID" 2022-11-23T01:42:15.9822500Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:42:15.9822705Z env: 2022-11-23T01:42:15.9822884Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:42:15.9823165Z DOCKER_CONTAINER_ID: 6e200e7a94c0c354a3a0a6fc7e5a55f6d42c9daee3c19a19715f86f8daa96d25 2022-11-23T01:42:15.9823437Z MONITOR_SCRIPT_PID: 764 2022-11-23T01:42:15.9823609Z ##[endgroup] 2022-11-23T01:42:15.9912161Z Prepare all required actions 2022-11-23T01:42:15.9912427Z Getting action download info 2022-11-23T01:42:16.1667628Z Download action repository 'actions/upload-artifact@v3' (SHA:83fd05a356d7e2593de66fc9913b3002723633cb) 2022-11-23T01:42:16.2992039Z ##[group]Run ./.github/actions/upload-test-artifacts 2022-11-23T01:42:16.2992262Z with: 2022-11-23T01:42:16.2992494Z file-suffix: test-distributed-2-2-linux.2xlarge_9655067063 2022-11-23T01:42:16.2992810Z env: 2022-11-23T01:42:16.2992982Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:42:16.2993270Z DOCKER_CONTAINER_ID: 6e200e7a94c0c354a3a0a6fc7e5a55f6d42c9daee3c19a19715f86f8daa96d25 2022-11-23T01:42:16.2993519Z ##[endgroup] 2022-11-23T01:42:16.3015635Z ##[group]Run # Remove any previous test jsons if they exist 2022-11-23T01:42:16.3015919Z # Remove any previous test jsons if they exist 2022-11-23T01:42:16.3016366Z rm -f test-jsons-*.zip 2022-11-23T01:42:16.3016650Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2022-11-23T01:42:16.3027733Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:42:16.3027949Z env: 2022-11-23T01:42:16.3028126Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:42:16.3028397Z DOCKER_CONTAINER_ID: 6e200e7a94c0c354a3a0a6fc7e5a55f6d42c9daee3c19a19715f86f8daa96d25 2022-11-23T01:42:16.3028729Z FILE_SUFFIX: test-distributed-2-2-linux.2xlarge_9655067063 2022-11-23T01:42:16.3028968Z ##[endgroup] 2022-11-23T01:42:16.3201428Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2022-11-23T01:42:16.3228113Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2022-11-23T01:42:16.3234030Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2022-11-23T01:42:16.3235111Z adding: test/.pytorch-slow-tests.json (deflated 73%) 2022-11-23T01:42:16.3243582Z adding: test/.pytorch-disabled-tests.json (deflated 86%) 2022-11-23T01:42:16.3281382Z ##[group]Run # Remove any previous test reports if they exist 2022-11-23T01:42:16.3281663Z # Remove any previous test reports if they exist 2022-11-23T01:42:16.3281899Z rm -f test-reports-*.zip 2022-11-23T01:42:16.3282161Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2022-11-23T01:42:16.3292736Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:42:16.3292951Z env: 2022-11-23T01:42:16.3293126Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:42:16.3293406Z DOCKER_CONTAINER_ID: 6e200e7a94c0c354a3a0a6fc7e5a55f6d42c9daee3c19a19715f86f8daa96d25 2022-11-23T01:42:16.3293723Z FILE_SUFFIX: test-distributed-2-2-linux.2xlarge_9655067063 2022-11-23T01:42:16.3294043Z ##[endgroup] 2022-11-23T01:42:16.3401306Z adding: test/test-reports/python-unittest/distributed.checkpoint.test_dedup_tensors/TEST-TestDedupTensor-20221123010436.xml (deflated 41%) 2022-11-23T01:42:16.3405387Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard/TEST-TestReshard-20221123010439.xml (deflated 63%) 2022-11-23T01:42:16.3409816Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding/TEST-TestShardedEmbedding-20221123010440.xml (deflated 61%) 2022-11-23T01:42:16.3421532Z adding: test/test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim/TEST-TestShardedOptimizer-20221123010442.xml (deflated 61%) 2022-11-23T01:42:16.3425475Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init/TEST-TestShardedTensorNNInit-20221123010443.xml (deflated 71%) 2022-11-23T01:42:16.3428731Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops/TEST-TestShardedTensorElementWiseOps-20221123010445.xml (deflated 76%) 2022-11-23T01:42:16.3431632Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops/TEST-TestTensorOps-20221123010446.xml (deflated 78%) 2022-11-23T01:42:16.3442375Z adding: test/test-reports/python-unittest/distributed._shard.sharding_plan.test_sharding_plan/TEST-TestShardingPlan-20221123010448.xml (deflated 78%) 2022-11-23T01:42:16.3446701Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_matrix_ops/TEST-TestShardedTensorMatrixOps-20221123010449.xml (deflated 88%) 2022-11-23T01:42:16.3450755Z adding: test/test-reports/python-unittest/distributed.test_dynamo_distributed/TEST-TestDistributed-20221123010451.xml (deflated 78%) 2022-11-23T01:42:16.3453666Z adding: test/test-reports/python-unittest/distributed.test_dynamo_distributed/TEST-TestDistributedMultiProc-20221123010451.xml (deflated 78%) 2022-11-23T01:42:16.3464881Z adding: test/test-reports/python-unittest/distributed._composable.test_checkpoint/TEST-TestCheckpoint-20221123010453.xml (deflated 55%) 2022-11-23T01:42:16.3468336Z adding: test/test-reports/python-unittest/distributed.elastic.metrics.api_test/TEST-MetricsApiTest-20221123010454.xml (deflated 63%) 2022-11-23T01:42:16.3472208Z adding: test/test-reports/python-unittest/distributed.nn.jit.test_instantiator/TEST-TestInstantiator-20221123010456.xml (deflated 64%) 2022-11-23T01:42:16.3476401Z adding: test/test-reports/python-unittest/distributed.checkpoint.test_utils/TEST-TestMedatadaIndex-20221123010458.xml (deflated 71%) 2022-11-23T01:42:16.3483934Z adding: test/test-reports/python-unittest/distributed.checkpoint.test_planner/TEST-TestSavePlan-20221123010500.xml (deflated 71%) 2022-11-23T01:42:16.3488100Z adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoad-20221123010501.xml (deflated 43%) 2022-11-23T01:42:16.3490994Z adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedReshardOnLoad-20221123010501.xml (deflated 74%) 2022-11-23T01:42:16.3494195Z adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20221123010501.xml (deflated 45%) 2022-11-23T01:42:16.3502428Z adding: test/test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper/TEST-CheckpointWrapperTest-20221123010503.xml (deflated 73%) 2022-11-23T01:42:16.3506987Z adding: test/test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-StoreUtilTest-20221123010505.xml (deflated 63%) 2022-11-23T01:42:16.3510598Z adding: test/test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-UtilTest-20221123010505.xml (deflated 69%) 2022-11-23T01:42:16.3514398Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20221123010507.xml (deflated 44%) 2022-11-23T01:42:16.3525821Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20221123010507.xml (deflated 43%) 2022-11-23T01:42:16.3529987Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven/TEST-TestUnevenParamShard-20221123010511.xml (deflated 41%) 2022-11-23T01:42:16.3533383Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_pure_fp16/TEST-TestPureFP16-20221123010514.xml (deflated 56%) 2022-11-23T01:42:16.3537137Z adding: test/test-reports/python-unittest/distributed._tensor.parallel.test_view_sharding_dim_change/TEST-TPViewShardingDimChangeTest-20221123010518.xml (deflated 43%) 2022-11-23T01:42:16.3548428Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_fx/TEST-TestSymbolicTracing-20221123010522.xml (deflated 45%) 2022-11-23T01:42:16.3552818Z adding: test/test-reports/python-unittest/distributed.test_c10d_error_logger/TEST-C10dErrorLoggerTest-20221123010526.xml (deflated 52%) 2022-11-23T01:42:16.3556552Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_memory/TEST-TestFSDPMemory-20221123010531.xml (deflated 59%) 2022-11-23T01:42:16.3562888Z adding: test/test-reports/python-unittest/distributed.checkpoint.test_checkpoint/TEST-TestDistributedFailure-20221123010536.xml (deflated 78%) 2022-11-23T01:42:16.3570145Z adding: test/test-reports/python-unittest/distributed.checkpoint.test_checkpoint/TEST-TestDistributedCheckpointing-20221123010536.xml (deflated 61%) 2022-11-23T01:42:16.3571321Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestClipGradNorm-20221123010541.xml (deflated 57%) 2022-11-23T01:42:16.3572483Z adding: test/test-reports/python-unittest/distributed._tensor.parallel.test_2d_parallel/TEST-Test2dParallelIntegration-20221123010547.xml (deflated 59%) 2022-11-23T01:42:16.3573577Z adding: test/test-reports/python-unittest/distributed._composable.test_replicate/TEST-ReplicateTest-20221123010552.xml (deflated 55%) 2022-11-23T01:42:16.3574598Z adding: test/test-reports/python-unittest/distributed._tensor.test_math_ops/TEST-DistMathOpsTest-20221123010558.xml (deflated 59%) 2022-11-23T01:42:16.3575679Z adding: test/test-reports/python-unittest/distributed._tensor.test_pointwise_ops/TEST-DistElementwiseOpsTest-20221123010605.xml (deflated 68%) 2022-11-23T01:42:16.3576830Z adding: test/test-reports/python-unittest/distributed._tensor.parallel.test_parallelize_api/TEST-TensorParallelAPITests-20221123010614.xml (deflated 70%) 2022-11-23T01:42:16.3578070Z adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedReshardOnLoad-20221123010624.xml (deflated 68%) 2022-11-23T01:42:16.3579362Z adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoad-20221123010624.xml (deflated 43%) 2022-11-23T01:42:16.3580811Z adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20221123010624.xml (deflated 45%) 2022-11-23T01:42:16.3582058Z adding: test/test-reports/python-unittest/distributed._tensor.test_api/TEST-DTensorAPITest-20221123010634.xml (deflated 71%) 2022-11-23T01:42:16.3583088Z adding: test/test-reports/python-unittest/distributed._tensor.test_tp_sharding_ops/TEST-TPShardingOpsTest-20221123010646.xml (deflated 74%) 2022-11-23T01:42:16.3584177Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_meta/TEST-TestFSDPWithMetaDevice-20221123010658.xml (deflated 87%) 2022-11-23T01:42:16.3585218Z adding: test/test-reports/python-unittest/distributed._tensor.test_matrix_ops/TEST-DistMatrixOpsTest-20221123010712.xml (deflated 74%) 2022-11-23T01:42:16.3586306Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardGradScaler-20221123010727.xml (deflated 68%) 2022-11-23T01:42:16.3587598Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardedGradScalerParityWithDDP-20221123010727.xml (deflated 85%) 2022-11-23T01:42:16.3588803Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights/TEST-TestFreezingWeights-20221123010743.xml (deflated 86%) 2022-11-23T01:42:16.3589995Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20221123010801.xml (deflated 44%) 2022-11-23T01:42:16.3591304Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20221123010803.xml (deflated 44%) 2022-11-23T01:42:16.3592663Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20221123010805.xml (deflated 44%) 2022-11-23T01:42:16.3593655Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20221123010807.xml (deflated 42%) 2022-11-23T01:42:16.3594705Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20221123010808.xml (deflated 42%) 2022-11-23T01:42:16.3595743Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20221123010810.xml (deflated 42%) 2022-11-23T01:42:16.3596533Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20221123010812.xml (deflated 42%) 2022-11-23T01:42:16.3597711Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123010814.xml (deflated 43%) 2022-11-23T01:42:16.3598891Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123010817.xml (deflated 44%) 2022-11-23T01:42:16.3600054Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123010821.xml (deflated 43%) 2022-11-23T01:42:16.3601101Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123010825.xml (deflated 43%) 2022-11-23T01:42:16.3601759Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123010829.xml (deflated 44%) 2022-11-23T01:42:16.3602820Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123010832.xml (deflated 43%) 2022-11-23T01:42:16.3603994Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123010836.xml (deflated 43%) 2022-11-23T01:42:16.3604823Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123010840.xml (deflated 43%) 2022-11-23T01:42:16.3605980Z adding: test/test-reports/python-unittest/distributed._tensor.parallel.test_tp_style/TEST-TensorParallelStyleTest-20221123010844.xml (deflated 79%) 2022-11-23T01:42:16.3606598Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123010903.xml (deflated 40%) 2022-11-23T01:42:16.3607270Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123010904.xml (deflated 40%) 2022-11-23T01:42:16.3608165Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123010906.xml (deflated 40%) 2022-11-23T01:42:16.3608684Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123010908.xml (deflated 40%) 2022-11-23T01:42:16.3609230Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20221123010910.xml (deflated 40%) 2022-11-23T01:42:16.3609747Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20221123010912.xml (deflated 40%) 2022-11-23T01:42:16.3610350Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20221123010914.xml (deflated 40%) 2022-11-23T01:42:16.3610907Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20221123010915.xml (deflated 40%) 2022-11-23T01:42:16.3611886Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixStoreTest-20221123010917.xml (deflated 41%) 2022-11-23T01:42:16.3612846Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20221123010919.xml (deflated 40%) 2022-11-23T01:42:16.3613575Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20221123010920.xml (deflated 40%) 2022-11-23T01:42:16.3614156Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20221123010922.xml (deflated 40%) 2022-11-23T01:42:16.3615071Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20221123010924.xml (deflated 40%) 2022-11-23T01:42:16.3615611Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20221123010926.xml (deflated 40%) 2022-11-23T01:42:16.3616151Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20221123010928.xml (deflated 40%) 2022-11-23T01:42:16.3616747Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123010929.xml (deflated 40%) 2022-11-23T01:42:16.3617297Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123010931.xml (deflated 40%) 2022-11-23T01:42:16.3617821Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123010933.xml (deflated 39%) 2022-11-23T01:42:16.3618356Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123010935.xml (deflated 40%) 2022-11-23T01:42:16.3619297Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20221123010947.xml (deflated 39%) 2022-11-23T01:42:16.3620240Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20221123010949.xml (deflated 39%) 2022-11-23T01:42:16.3621159Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123010950.xml (deflated 39%) 2022-11-23T01:42:16.3622088Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123010952.xml (deflated 39%) 2022-11-23T01:42:16.3623007Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123010954.xml (deflated 39%) 2022-11-23T01:42:16.3623931Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123010956.xml (deflated 38%) 2022-11-23T01:42:16.3624826Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123010958.xml (deflated 38%) 2022-11-23T01:42:16.3625743Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123010959.xml (deflated 39%) 2022-11-23T01:42:16.3626661Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123011001.xml (deflated 40%) 2022-11-23T01:42:16.3627574Z adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123011005.xml (deflated 39%) 2022-11-23T01:42:16.3628615Z adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20221123011007.xml (deflated 55%) 2022-11-23T01:42:16.3629809Z adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20221123011007.xml (deflated 80%) 2022-11-23T01:42:16.3630990Z adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20221123011007.xml (deflated 79%) 2022-11-23T01:42:16.3632102Z adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20221123011007.xml (deflated 64%) 2022-11-23T01:42:16.3633204Z adding: test/test-reports/python-unittest/distributed._tensor.test_view_ops/TEST-TestViewOps-20221123011029.xml (deflated 53%) 2022-11-23T01:42:16.3634263Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks/TEST-TestCommunicationHooks-20221123011058.xml (deflated 92%) 2022-11-23T01:42:16.3635370Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint/TEST-TestFSDPCheckpoint-20221123011127.xml (deflated 91%) 2022-11-23T01:42:16.3636528Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsFQNs-20221123011157.xml (deflated 56%) 2022-11-23T01:42:16.3637803Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsMultipleParamGroups-20221123011157.xml (deflated 85%) 2022-11-23T01:42:16.3639136Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsParamAccess-20221123011157.xml (deflated 45%) 2022-11-23T01:42:16.3640422Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsUnshardReshard-20221123011157.xml (deflated 78%) 2022-11-23T01:42:16.3641897Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsWriteback-20221123011157.xml (deflated 67%) 2022-11-23T01:42:16.3643221Z adding: test/test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerDistributed-20221123011229.xml (deflated 91%) 2022-11-23T01:42:16.3644565Z adding: test/test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerSingleRank-20221123011229.xml (deflated 73%) 2022-11-23T01:42:16.3645779Z adding: test/test-reports/python-unittest/distributed._tensor.test_device_mesh/TEST-DeviceMeshCollectiveTest-20221123011311.xml (deflated 84%) 2022-11-23T01:42:16.3646836Z adding: test/test-reports/python-unittest/distributed._tensor.test_device_mesh/TEST-DeviceMeshTest-20221123011311.xml (deflated 68%) 2022-11-23T01:42:16.3647962Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionSharded-20221123011409.xml (deflated 93%) 2022-11-23T01:42:16.3649163Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionUnsharded-20221123011409.xml (deflated 68%) 2022-11-23T01:42:16.3650355Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParams-20221123011534.xml (deflated 93%) 2022-11-23T01:42:16.3651521Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParamsNoShard-20221123011534.xml (deflated 43%) 2022-11-23T01:42:16.3652765Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state/TEST-TestFSDPOptimState-20221123011700.xml (deflated 93%) 2022-11-23T01:42:16.3653769Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestHooks-20221123011836.xml (deflated 81%) 2022-11-23T01:42:16.3654718Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestNoGrad-20221123011836.xml (deflated 64%) 2022-11-23T01:42:16.3655685Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParamInit-20221123011836.xml (deflated 62%) 2022-11-23T01:42:16.3656707Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParityWithDDP-20221123011836.xml (deflated 93%) 2022-11-23T01:42:16.3657752Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict/TEST-TestFSDPStateDict-20221123012013.xml (deflated 96%) 2022-11-23T01:42:16.3658919Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentDistAutogradTest-20221123012324.xml (deflated 43%) 2022-11-23T01:42:16.3660190Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentDistAutogradTest-20221123012336.xml (deflated 42%) 2022-11-23T01:42:16.3661370Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012346.xml (deflated 41%) 2022-11-23T01:42:16.3662470Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012356.xml (deflated 41%) 2022-11-23T01:42:16.3663563Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012406.xml (deflated 41%) 2022-11-23T01:42:16.3664650Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012421.xml (deflated 42%) 2022-11-23T01:42:16.3665759Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012431.xml (deflated 41%) 2022-11-23T01:42:16.3666865Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012436.xml (deflated 42%) 2022-11-23T01:42:16.3667953Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012446.xml (deflated 41%) 2022-11-23T01:42:16.3669075Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012455.xml (deflated 41%) 2022-11-23T01:42:16.3670185Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012501.xml (deflated 41%) 2022-11-23T01:42:16.3671280Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012511.xml (deflated 41%) 2022-11-23T01:42:16.3672378Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012522.xml (deflated 41%) 2022-11-23T01:42:16.3673522Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012531.xml (deflated 41%) 2022-11-23T01:42:16.3674620Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012537.xml (deflated 41%) 2022-11-23T01:42:16.3675732Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012555.xml (deflated 41%) 2022-11-23T01:42:16.3676859Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012606.xml (deflated 41%) 2022-11-23T01:42:16.3677934Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012613.xml (deflated 41%) 2022-11-23T01:42:16.3679025Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012623.xml (deflated 41%) 2022-11-23T01:42:16.3680122Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012630.xml (deflated 41%) 2022-11-23T01:42:16.3681292Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012640.xml (deflated 41%) 2022-11-23T01:42:16.3682429Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20221123012650.xml (deflated 41%) 2022-11-23T01:42:16.3683570Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20221123012700.xml (deflated 41%) 2022-11-23T01:42:16.3684756Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20221123012711.xml (deflated 41%) 2022-11-23T01:42:16.3685934Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20221123012721.xml (deflated 42%) 2022-11-23T01:42:16.3687183Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20221123012731.xml (deflated 41%) 2022-11-23T01:42:16.3688361Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20221123012737.xml (deflated 41%) 2022-11-23T01:42:16.3689519Z adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20221123012746.xml (deflated 41%) 2022-11-23T01:42:16.3690784Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDdpComparisonTest-20221123012757.xml (deflated 41%) 2022-11-23T01:42:16.3692087Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221123012758.xml (deflated 43%) 2022-11-23T01:42:16.3693353Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221123012803.xml (deflated 42%) 2022-11-23T01:42:16.3694644Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221123012807.xml (deflated 42%) 2022-11-23T01:42:16.3695974Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221123012811.xml (deflated 43%) 2022-11-23T01:42:16.3697267Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221123012815.xml (deflated 42%) 2022-11-23T01:42:16.3698513Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221123012819.xml (deflated 42%) 2022-11-23T01:42:16.3699751Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221123012822.xml (deflated 43%) 2022-11-23T01:42:16.3700957Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRpcTest-20221123012826.xml (deflated 42%) 2022-11-23T01:42:16.3702166Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123012830.xml (deflated 41%) 2022-11-23T01:42:16.3703392Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123012834.xml (deflated 41%) 2022-11-23T01:42:16.3704606Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123012838.xml (deflated 42%) 2022-11-23T01:42:16.3705816Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123012842.xml (deflated 41%) 2022-11-23T01:42:16.3707026Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123012846.xml (deflated 41%) 2022-11-23T01:42:16.3708199Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123012848.xml (deflated 41%) 2022-11-23T01:42:16.3708848Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123012850.xml (deflated 41%) 2022-11-23T01:42:16.3709513Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123012851.xml (deflated 41%) 2022-11-23T01:42:16.3710220Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012853.xml (deflated 43%) 2022-11-23T01:42:16.3710958Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012857.xml (deflated 43%) 2022-11-23T01:42:16.3711747Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012901.xml (deflated 44%) 2022-11-23T01:42:16.3712759Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012905.xml (deflated 45%) 2022-11-23T01:42:16.3713579Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012909.xml (deflated 44%) 2022-11-23T01:42:16.3714319Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012913.xml (deflated 44%) 2022-11-23T01:42:16.3715065Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012917.xml (deflated 44%) 2022-11-23T01:42:16.3715782Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012921.xml (deflated 44%) 2022-11-23T01:42:16.3716521Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012926.xml (deflated 44%) 2022-11-23T01:42:16.3717299Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012930.xml (deflated 45%) 2022-11-23T01:42:16.3718038Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012934.xml (deflated 44%) 2022-11-23T01:42:16.3718758Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012938.xml (deflated 44%) 2022-11-23T01:42:16.3719483Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012942.xml (deflated 44%) 2022-11-23T01:42:16.3720214Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012946.xml (deflated 44%) 2022-11-23T01:42:16.3721048Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012950.xml (deflated 44%) 2022-11-23T01:42:16.3721777Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012954.xml (deflated 44%) 2022-11-23T01:42:16.3722495Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123012958.xml (deflated 43%) 2022-11-23T01:42:16.3723228Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013000.xml (deflated 44%) 2022-11-23T01:42:16.3723963Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013004.xml (deflated 44%) 2022-11-23T01:42:16.3724697Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013009.xml (deflated 44%) 2022-11-23T01:42:16.3725420Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013013.xml (deflated 44%) 2022-11-23T01:42:16.3726152Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013017.xml (deflated 44%) 2022-11-23T01:42:16.3726883Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013021.xml (deflated 44%) 2022-11-23T01:42:16.3727673Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013025.xml (deflated 44%) 2022-11-23T01:42:16.3728410Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013029.xml (deflated 43%) 2022-11-23T01:42:16.3729132Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013031.xml (deflated 44%) 2022-11-23T01:42:16.3729866Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013036.xml (deflated 44%) 2022-11-23T01:42:16.3730598Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013040.xml (deflated 44%) 2022-11-23T01:42:16.3731332Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013044.xml (deflated 44%) 2022-11-23T01:42:16.3732049Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013048.xml (deflated 44%) 2022-11-23T01:42:16.3732810Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013052.xml (deflated 44%) 2022-11-23T01:42:16.3733545Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013056.xml (deflated 44%) 2022-11-23T01:42:16.3734271Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013100.xml (deflated 44%) 2022-11-23T01:42:16.3735002Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013104.xml (deflated 44%) 2022-11-23T01:42:16.3735719Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013108.xml (deflated 44%) 2022-11-23T01:42:16.3736446Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013112.xml (deflated 44%) 2022-11-23T01:42:16.3737178Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013116.xml (deflated 44%) 2022-11-23T01:42:16.3737903Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013121.xml (deflated 44%) 2022-11-23T01:42:16.3738619Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013125.xml (deflated 43%) 2022-11-23T01:42:16.3739346Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013129.xml (deflated 44%) 2022-11-23T01:42:16.3740075Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013133.xml (deflated 44%) 2022-11-23T01:42:16.3740806Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013137.xml (deflated 44%) 2022-11-23T01:42:16.3741521Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013141.xml (deflated 44%) 2022-11-23T01:42:16.3742239Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013145.xml (deflated 44%) 2022-11-23T01:42:16.3742965Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013149.xml (deflated 44%) 2022-11-23T01:42:16.3743732Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013153.xml (deflated 44%) 2022-11-23T01:42:16.3744460Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013157.xml (deflated 44%) 2022-11-23T01:42:16.3745177Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013201.xml (deflated 45%) 2022-11-23T01:42:16.3745902Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013205.xml (deflated 45%) 2022-11-23T01:42:16.3746626Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013210.xml (deflated 45%) 2022-11-23T01:42:16.3747355Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013214.xml (deflated 44%) 2022-11-23T01:42:16.3748065Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013218.xml (deflated 44%) 2022-11-23T01:42:16.3748914Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013222.xml (deflated 44%) 2022-11-23T01:42:16.3749644Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013226.xml (deflated 44%) 2022-11-23T01:42:16.3750358Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013230.xml (deflated 44%) 2022-11-23T01:42:16.3751096Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013234.xml (deflated 43%) 2022-11-23T01:42:16.3751825Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013238.xml (deflated 44%) 2022-11-23T01:42:16.3752558Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013242.xml (deflated 44%) 2022-11-23T01:42:16.3753346Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013246.xml (deflated 44%) 2022-11-23T01:42:16.3754079Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013250.xml (deflated 44%) 2022-11-23T01:42:16.3754810Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013254.xml (deflated 44%) 2022-11-23T01:42:16.3755549Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013258.xml (deflated 44%) 2022-11-23T01:42:16.3756283Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013302.xml (deflated 44%) 2022-11-23T01:42:16.3757002Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013306.xml (deflated 43%) 2022-11-23T01:42:16.3757736Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013310.xml (deflated 44%) 2022-11-23T01:42:16.3758467Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013315.xml (deflated 44%) 2022-11-23T01:42:16.3759230Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013319.xml (deflated 44%) 2022-11-23T01:42:16.3759944Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013323.xml (deflated 44%) 2022-11-23T01:42:16.3760772Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013327.xml (deflated 43%) 2022-11-23T01:42:16.3761510Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013331.xml (deflated 43%) 2022-11-23T01:42:16.3762247Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013335.xml (deflated 43%) 2022-11-23T01:42:16.3762979Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013339.xml (deflated 43%) 2022-11-23T01:42:16.3763694Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013343.xml (deflated 43%) 2022-11-23T01:42:16.3764474Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013347.xml (deflated 43%) 2022-11-23T01:42:16.3765206Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013351.xml (deflated 43%) 2022-11-23T01:42:16.3765931Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013353.xml (deflated 43%) 2022-11-23T01:42:16.3766644Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013357.xml (deflated 43%) 2022-11-23T01:42:16.3767371Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013401.xml (deflated 43%) 2022-11-23T01:42:16.3768102Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013406.xml (deflated 43%) 2022-11-23T01:42:16.3768834Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013410.xml (deflated 43%) 2022-11-23T01:42:16.3769549Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013414.xml (deflated 43%) 2022-11-23T01:42:16.3770279Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013418.xml (deflated 43%) 2022-11-23T01:42:16.3771013Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013422.xml (deflated 43%) 2022-11-23T01:42:16.3771748Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013426.xml (deflated 43%) 2022-11-23T01:42:16.3772475Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013430.xml (deflated 43%) 2022-11-23T01:42:16.3773188Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013434.xml (deflated 44%) 2022-11-23T01:42:16.3773912Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123013438.xml (deflated 44%) 2022-11-23T01:42:16.3774703Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221123013442.xml (deflated 44%) 2022-11-23T01:42:16.3775466Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221123013446.xml (deflated 45%) 2022-11-23T01:42:16.3776222Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221123013450.xml (deflated 44%) 2022-11-23T01:42:16.3776839Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123013456.xml (deflated 39%) 2022-11-23T01:42:16.3777346Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123013459.xml (deflated 40%) 2022-11-23T01:42:16.3777847Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123013503.xml (deflated 39%) 2022-11-23T01:42:16.3778337Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123013506.xml (deflated 39%) 2022-11-23T01:42:16.3778837Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123013510.xml (deflated 39%) 2022-11-23T01:42:16.3779337Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123013513.xml (deflated 40%) 2022-11-23T01:42:16.3779865Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123013517.xml (deflated 39%) 2022-11-23T01:42:16.3780353Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123013520.xml (deflated 40%) 2022-11-23T01:42:16.3780848Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123013524.xml (deflated 39%) 2022-11-23T01:42:16.3781338Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123013527.xml (deflated 40%) 2022-11-23T01:42:16.3781835Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123013531.xml (deflated 40%) 2022-11-23T01:42:16.3782334Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123013534.xml (deflated 38%) 2022-11-23T01:42:16.3782856Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123013538.xml (deflated 39%) 2022-11-23T01:42:16.3783372Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123013541.xml (deflated 38%) 2022-11-23T01:42:16.3783887Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123013545.xml (deflated 39%) 2022-11-23T01:42:16.3784386Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123013548.xml (deflated 38%) 2022-11-23T01:42:16.3784904Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123013552.xml (deflated 40%) 2022-11-23T01:42:16.3785425Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123013555.xml (deflated 39%) 2022-11-23T01:42:16.3785940Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123013559.xml (deflated 40%) 2022-11-23T01:42:16.3786440Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123013602.xml (deflated 38%) 2022-11-23T01:42:16.3786958Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123013606.xml (deflated 39%) 2022-11-23T01:42:16.3787471Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123013609.xml (deflated 40%) 2022-11-23T01:42:16.3788037Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013613.xml (deflated 44%) 2022-11-23T01:42:16.3788639Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013616.xml (deflated 45%) 2022-11-23T01:42:16.3789274Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013620.xml (deflated 43%) 2022-11-23T01:42:16.3789884Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013623.xml (deflated 43%) 2022-11-23T01:42:16.3790496Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013627.xml (deflated 45%) 2022-11-23T01:42:16.3791091Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013630.xml (deflated 45%) 2022-11-23T01:42:16.3791696Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013634.xml (deflated 46%) 2022-11-23T01:42:16.3792303Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013637.xml (deflated 46%) 2022-11-23T01:42:16.3792984Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013641.xml (deflated 44%) 2022-11-23T01:42:16.3793577Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013644.xml (deflated 45%) 2022-11-23T01:42:16.3794225Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013648.xml (deflated 45%) 2022-11-23T01:42:16.3794830Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013651.xml (deflated 44%) 2022-11-23T01:42:16.3795436Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013655.xml (deflated 44%) 2022-11-23T01:42:16.3796030Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013658.xml (deflated 44%) 2022-11-23T01:42:16.3796635Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013702.xml (deflated 44%) 2022-11-23T01:42:16.3797239Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013705.xml (deflated 45%) 2022-11-23T01:42:16.3797850Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013709.xml (deflated 44%) 2022-11-23T01:42:16.3798440Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013712.xml (deflated 46%) 2022-11-23T01:42:16.3799048Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013716.xml (deflated 45%) 2022-11-23T01:42:16.3799646Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013719.xml (deflated 50%) 2022-11-23T01:42:16.3800256Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013723.xml (deflated 42%) 2022-11-23T01:42:16.3800936Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013726.xml (deflated 42%) 2022-11-23T01:42:16.3801544Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013730.xml (deflated 42%) 2022-11-23T01:42:16.3802141Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013733.xml (deflated 42%) 2022-11-23T01:42:16.3802748Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013737.xml (deflated 43%) 2022-11-23T01:42:16.3803337Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013740.xml (deflated 42%) 2022-11-23T01:42:16.3803992Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013744.xml (deflated 42%) 2022-11-23T01:42:16.3804590Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013747.xml (deflated 41%) 2022-11-23T01:42:16.3805197Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013751.xml (deflated 41%) 2022-11-23T01:42:16.3805803Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013754.xml (deflated 44%) 2022-11-23T01:42:16.3806389Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013758.xml (deflated 45%) 2022-11-23T01:42:16.3806988Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013801.xml (deflated 42%) 2022-11-23T01:42:16.3807699Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013805.xml (deflated 41%) 2022-11-23T01:42:16.3808302Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013808.xml (deflated 42%) 2022-11-23T01:42:16.3808936Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013811.xml (deflated 42%) 2022-11-23T01:42:16.3809540Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013815.xml (deflated 42%) 2022-11-23T01:42:16.3810141Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013818.xml (deflated 42%) 2022-11-23T01:42:16.3810740Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123013822.xml (deflated 42%) 2022-11-23T01:42:16.3811418Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20221123013825.xml (deflated 43%) 2022-11-23T01:42:16.3812168Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20221123013829.xml (deflated 43%) 2022-11-23T01:42:16.3812906Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20221123013832.xml (deflated 44%) 2022-11-23T01:42:16.3813643Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20221123013836.xml (deflated 42%) 2022-11-23T01:42:16.3814278Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013839.xml (deflated 40%) 2022-11-23T01:42:16.3814845Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013843.xml (deflated 41%) 2022-11-23T01:42:16.3815391Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123014148.xml (deflated 39%) 2022-11-23T01:42:16.3815941Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013846.xml (deflated 39%) 2022-11-23T01:42:16.3816497Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013850.xml (deflated 40%) 2022-11-23T01:42:16.3817065Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013854.xml (deflated 39%) 2022-11-23T01:42:16.3817639Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013857.xml (deflated 40%) 2022-11-23T01:42:16.3818196Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013901.xml (deflated 40%) 2022-11-23T01:42:16.3818747Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013906.xml (deflated 41%) 2022-11-23T01:42:16.3819357Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013910.xml (deflated 40%) 2022-11-23T01:42:16.3819925Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013914.xml (deflated 41%) 2022-11-23T01:42:16.3820495Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013917.xml (deflated 40%) 2022-11-23T01:42:16.3821043Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013921.xml (deflated 40%) 2022-11-23T01:42:16.3821610Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013925.xml (deflated 40%) 2022-11-23T01:42:16.3822169Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013928.xml (deflated 40%) 2022-11-23T01:42:16.3822738Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013932.xml (deflated 40%) 2022-11-23T01:42:16.3823290Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013936.xml (deflated 40%) 2022-11-23T01:42:16.3823888Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013940.xml (deflated 40%) 2022-11-23T01:42:16.3824456Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013943.xml (deflated 40%) 2022-11-23T01:42:16.3825016Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013948.xml (deflated 40%) 2022-11-23T01:42:16.3825561Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013952.xml (deflated 41%) 2022-11-23T01:42:16.3826118Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123013956.xml (deflated 40%) 2022-11-23T01:42:16.3826683Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014000.xml (deflated 41%) 2022-11-23T01:42:16.3827244Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014004.xml (deflated 41%) 2022-11-23T01:42:16.3827795Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014007.xml (deflated 40%) 2022-11-23T01:42:16.3828355Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014011.xml (deflated 41%) 2022-11-23T01:42:16.3828919Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014015.xml (deflated 41%) 2022-11-23T01:42:16.3829475Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014019.xml (deflated 40%) 2022-11-23T01:42:16.3830023Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014022.xml (deflated 40%) 2022-11-23T01:42:16.3830588Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014026.xml (deflated 41%) 2022-11-23T01:42:16.3831149Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014030.xml (deflated 40%) 2022-11-23T01:42:16.3831707Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014033.xml (deflated 40%) 2022-11-23T01:42:16.3832251Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014037.xml (deflated 40%) 2022-11-23T01:42:16.3832878Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014042.xml (deflated 41%) 2022-11-23T01:42:16.3833484Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014046.xml (deflated 40%) 2022-11-23T01:42:16.3834052Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014050.xml (deflated 40%) 2022-11-23T01:42:16.3834605Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014053.xml (deflated 41%) 2022-11-23T01:42:16.3835171Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014057.xml (deflated 40%) 2022-11-23T01:42:16.3835738Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014101.xml (deflated 40%) 2022-11-23T01:42:16.3836301Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014106.xml (deflated 41%) 2022-11-23T01:42:16.3836848Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014110.xml (deflated 40%) 2022-11-23T01:42:16.3837407Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014113.xml (deflated 40%) 2022-11-23T01:42:16.3837966Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014117.xml (deflated 40%) 2022-11-23T01:42:16.3838558Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014121.xml (deflated 41%) 2022-11-23T01:42:16.3839106Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014125.xml (deflated 40%) 2022-11-23T01:42:16.3839664Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014129.xml (deflated 40%) 2022-11-23T01:42:16.3840225Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014134.xml (deflated 41%) 2022-11-23T01:42:16.3840882Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014135.xml (deflated 40%) 2022-11-23T01:42:16.3841435Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014139.xml (deflated 42%) 2022-11-23T01:42:16.3842004Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014141.xml (deflated 41%) 2022-11-23T01:42:16.3842569Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123014144.xml (deflated 40%) 2022-11-23T01:42:16.3843110Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123014149.xml (deflated 39%) 2022-11-23T01:42:16.3843614Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123014151.xml (deflated 39%) 2022-11-23T01:42:16.3844134Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123014152.xml (deflated 40%) 2022-11-23T01:42:16.3844648Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123014154.xml (deflated 39%) 2022-11-23T01:42:16.3845157Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123014155.xml (deflated 40%) 2022-11-23T01:42:16.3845685Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-RendezvousEnvTest-20221123014157.xml (deflated 39%) 2022-11-23T01:42:16.3846220Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-TimeoutTest-20221123014159.xml (deflated 41%) 2022-11-23T01:42:16.3862488Z ##[group]Run # Remove any previous test reports if they exist 2022-11-23T01:42:16.3862779Z # Remove any previous test reports if they exist 2022-11-23T01:42:16.3863009Z rm -f usage-log-*.zip 2022-11-23T01:42:16.3863282Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2022-11-23T01:42:16.3863633Z # so check to see if the file exists first 2022-11-23T01:42:16.3863848Z if [ -f 'usage_log.txt' ]; then 2022-11-23T01:42:16.3864093Z  zip "usage-log-${FILE_SUFFIX}.zip" 'usage_log.txt' 2022-11-23T01:42:16.3864308Z fi 2022-11-23T01:42:16.3875271Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:42:16.3875496Z env: 2022-11-23T01:42:16.3875672Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:42:16.3875939Z DOCKER_CONTAINER_ID: 6e200e7a94c0c354a3a0a6fc7e5a55f6d42c9daee3c19a19715f86f8daa96d25 2022-11-23T01:42:16.3876268Z FILE_SUFFIX: test-distributed-2-2-linux.2xlarge_9655067063 2022-11-23T01:42:16.3876505Z ##[endgroup] 2022-11-23T01:42:16.4184824Z adding: usage_log.txt (deflated 95%) 2022-11-23T01:42:16.4233416Z ##[group]Run seemethere/upload-artifact-s3@v5 2022-11-23T01:42:16.4233627Z with: 2022-11-23T01:42:16.4233816Z s3-prefix: pytorch/pytorch/3528293562/1/artifact 2022-11-23T01:42:16.4234041Z retention-days: 14 2022-11-23T01:42:16.4234234Z if-no-files-found: warn 2022-11-23T01:42:16.4234417Z path: test-jsons-*.zip 2022-11-23T01:42:16.4234601Z name: artifact 2022-11-23T01:42:16.4234782Z s3-bucket: gha-artifacts 2022-11-23T01:42:16.4234956Z region: us-east-1 2022-11-23T01:42:16.4235118Z env: 2022-11-23T01:42:16.4235286Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:42:16.4235659Z DOCKER_CONTAINER_ID: 6e200e7a94c0c354a3a0a6fc7e5a55f6d42c9daee3c19a19715f86f8daa96d25 2022-11-23T01:42:16.4235921Z ##[endgroup] 2022-11-23T01:42:16.8144974Z NOTE: s3-prefix specified, ignoring name parameter 2022-11-23T01:42:16.8145604Z With the provided path, there will be 1 file uploaded 2022-11-23T01:42:16.8146011Z Uploading to s3 prefix: pytorch/pytorch/3528293562/1/artifact 2022-11-23T01:42:16.8206459Z Starting upload of test-jsons-test-distributed-2-2-linux.2xlarge_9655067063.zip 2022-11-23T01:42:17.0058159Z Finished upload of test-jsons-test-distributed-2-2-linux.2xlarge_9655067063.zip 2022-11-23T01:42:17.0203413Z ##[group]Run seemethere/upload-artifact-s3@v5 2022-11-23T01:42:17.0203629Z with: 2022-11-23T01:42:17.0203817Z s3-prefix: pytorch/pytorch/3528293562/1/artifact 2022-11-23T01:42:17.0204036Z retention-days: 14 2022-11-23T01:42:17.0204239Z if-no-files-found: error 2022-11-23T01:42:17.0204429Z path: test-reports-*.zip 2022-11-23T01:42:17.0204612Z name: artifact 2022-11-23T01:42:17.0204799Z s3-bucket: gha-artifacts 2022-11-23T01:42:17.0204973Z region: us-east-1 2022-11-23T01:42:17.0205138Z env: 2022-11-23T01:42:17.0205310Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:42:17.0205576Z DOCKER_CONTAINER_ID: 6e200e7a94c0c354a3a0a6fc7e5a55f6d42c9daee3c19a19715f86f8daa96d25 2022-11-23T01:42:17.0205839Z ##[endgroup] 2022-11-23T01:42:17.3565955Z NOTE: s3-prefix specified, ignoring name parameter 2022-11-23T01:42:17.3566420Z With the provided path, there will be 1 file uploaded 2022-11-23T01:42:17.3566897Z Uploading to s3 prefix: pytorch/pytorch/3528293562/1/artifact 2022-11-23T01:42:17.3574065Z Starting upload of test-reports-test-distributed-2-2-linux.2xlarge_9655067063.zip 2022-11-23T01:42:17.5172211Z Finished upload of test-reports-test-distributed-2-2-linux.2xlarge_9655067063.zip 2022-11-23T01:42:17.5310519Z ##[group]Run seemethere/upload-artifact-s3@v5 2022-11-23T01:42:17.5310740Z with: 2022-11-23T01:42:17.5310932Z s3-prefix: pytorch/pytorch/3528293562/1/artifact 2022-11-23T01:42:17.5311158Z retention-days: 14 2022-11-23T01:42:17.5311353Z if-no-files-found: ignore 2022-11-23T01:42:17.5311539Z path: usage-log-*.zip 2022-11-23T01:42:17.5311720Z name: artifact 2022-11-23T01:42:17.5311907Z s3-bucket: gha-artifacts 2022-11-23T01:42:17.5312084Z region: us-east-1 2022-11-23T01:42:17.5312247Z env: 2022-11-23T01:42:17.5312416Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:42:17.5312777Z DOCKER_CONTAINER_ID: 6e200e7a94c0c354a3a0a6fc7e5a55f6d42c9daee3c19a19715f86f8daa96d25 2022-11-23T01:42:17.5313038Z ##[endgroup] 2022-11-23T01:42:17.8679543Z NOTE: s3-prefix specified, ignoring name parameter 2022-11-23T01:42:17.8680360Z With the provided path, there will be 1 file uploaded 2022-11-23T01:42:17.8680987Z Uploading to s3 prefix: pytorch/pytorch/3528293562/1/artifact 2022-11-23T01:42:17.8688859Z Starting upload of usage-log-test-distributed-2-2-linux.2xlarge_9655067063.zip 2022-11-23T01:42:18.0621244Z Finished upload of usage-log-test-distributed-2-2-linux.2xlarge_9655067063.zip 2022-11-23T01:42:18.0757744Z ##[group]Run # shellcheck disable=SC2156 2022-11-23T01:42:18.0758000Z # shellcheck disable=SC2156 2022-11-23T01:42:18.0758297Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2022-11-23T01:42:18.0769997Z shell: /usr/bin/bash -e {0} 2022-11-23T01:42:18.0770187Z env: 2022-11-23T01:42:18.0770361Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:42:18.0770631Z DOCKER_CONTAINER_ID: 6e200e7a94c0c354a3a0a6fc7e5a55f6d42c9daee3c19a19715f86f8daa96d25 2022-11-23T01:42:18.0770894Z ##[endgroup] 2022-11-23T01:42:19.8188100Z ##[group]Run set -x 2022-11-23T01:42:19.8188303Z set -x 2022-11-23T01:42:19.8188510Z python3 -m pip install -r requirements.txt 2022-11-23T01:42:19.8188757Z python3 -m pip install boto3==1.19.12 2022-11-23T01:42:19.8189049Z python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test 2022-11-23T01:42:19.8200318Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:42:19.8200540Z env: 2022-11-23T01:42:19.8200994Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:42:19.8201276Z DOCKER_CONTAINER_ID: 6e200e7a94c0c354a3a0a6fc7e5a55f6d42c9daee3c19a19715f86f8daa96d25 2022-11-23T01:42:19.8201541Z AWS_DEFAULT_REGION: us-east-1 2022-11-23T01:42:19.8201730Z BRANCH: master 2022-11-23T01:42:19.8201910Z TEST_CONFIG: distributed 2022-11-23T01:42:19.8202080Z SHARD_NUMBER: 2 2022-11-23T01:42:19.8202292Z BUILD_ENVIRONMENT: linux-focal-py3.7-gcc7 2022-11-23T01:42:19.8202501Z PR_NUMBER: 2022-11-23T01:42:19.8202711Z PYTORCH_RETRY_TEST_CASES: 1 2022-11-23T01:42:19.8202917Z PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1 2022-11-23T01:42:19.8203126Z SHA1: 1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T01:42:19.8203317Z TAG: 2022-11-23T01:42:19.8203483Z WORKFLOW_ID: 3528293562 2022-11-23T01:42:19.8203807Z GITHUB_TOKEN: *** 2022-11-23T01:42:19.8203996Z GHA_WORKFLOW_JOB_ID: 9655067063 2022-11-23T01:42:19.8204183Z ##[endgroup] 2022-11-23T01:42:19.8228571Z + python3 -m pip install -r requirements.txt 2022-11-23T01:42:20.0334890Z Defaulting to user installation because normal site-packages is not writeable 2022-11-23T01:42:20.0614756Z Requirement already satisfied: astunparse in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 2)) (1.6.3) 2022-11-23T01:42:20.0642541Z Requirement already satisfied: expecttest in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 3)) (0.1.4) 2022-11-23T01:42:20.0649906Z Requirement already satisfied: future in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (0.18.2) 2022-11-23T01:42:20.0657830Z Requirement already satisfied: hypothesis in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 5)) (6.58.0) 2022-11-23T01:42:20.1023070Z Requirement already satisfied: numpy in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 6)) (1.21.6) 2022-11-23T01:42:20.1031453Z Requirement already satisfied: psutil in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 7)) (5.9.1) 2022-11-23T01:42:20.1107164Z Requirement already satisfied: pyyaml in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 8)) (6.0) 2022-11-23T01:42:20.1114770Z Requirement already satisfied: requests in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 9)) (2.26.0) 2022-11-23T01:42:20.1328659Z Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from -r requirements.txt (line 10)) (49.1.3) 2022-11-23T01:42:20.1494281Z Requirement already satisfied: six in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 11)) (1.16.0) 2022-11-23T01:42:20.1502402Z Requirement already satisfied: types-dataclasses in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 12)) (0.6.6) 2022-11-23T01:42:20.1507811Z Requirement already satisfied: typing_extensions in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 13)) (4.4.0) 2022-11-23T01:42:20.1517559Z Requirement already satisfied: sympy in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 14)) (1.10.1) 2022-11-23T01:42:20.1535980Z Requirement already satisfied: filelock in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 15)) (3.8.0) 2022-11-23T01:42:20.1606354Z Requirement already satisfied: networkx in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 16)) (2.6.3) 2022-11-23T01:42:20.1765120Z Requirement already satisfied: jinja2 in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 17)) (3.1.2) 2022-11-23T01:42:20.1787014Z Requirement already satisfied: wheel<1.0,>=0.23.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from astunparse->-r requirements.txt (line 2)) (0.38.4) 2022-11-23T01:42:20.1804549Z Requirement already satisfied: sortedcontainers<3.0.0,>=2.1.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (2.4.0) 2022-11-23T01:42:20.1814137Z Requirement already satisfied: exceptiongroup>=1.0.0; python_version < "3.11" in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (1.0.4) 2022-11-23T01:42:20.1829906Z Requirement already satisfied: attrs>=19.2.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (22.1.0) 2022-11-23T01:42:20.2079883Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (1.26.12) 2022-11-23T01:42:20.2233372Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2022.9.24) 2022-11-23T01:42:20.2242864Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2.0.12) 2022-11-23T01:42:20.2261186Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (3.4) 2022-11-23T01:42:20.2272536Z Requirement already satisfied: mpmath>=0.19 in /home/ec2-user/.local/lib/python3.7/site-packages (from sympy->-r requirements.txt (line 14)) (1.2.1) 2022-11-23T01:42:20.2326679Z Requirement already satisfied: MarkupSafe>=2.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from jinja2->-r requirements.txt (line 17)) (2.1.1) 2022-11-23T01:42:20.2921376Z + python3 -m pip install boto3==1.19.12 2022-11-23T01:42:20.4994996Z Defaulting to user installation because normal site-packages is not writeable 2022-11-23T01:42:20.5175697Z Requirement already satisfied: boto3==1.19.12 in /home/ec2-user/.local/lib/python3.7/site-packages (1.19.12) 2022-11-23T01:42:20.5223455Z Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.10.0) 2022-11-23T01:42:20.5235605Z Requirement already satisfied: botocore<1.23.0,>=1.22.12 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (1.22.12) 2022-11-23T01:42:20.5276366Z Requirement already satisfied: s3transfer<0.6.0,>=0.5.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.5.2) 2022-11-23T01:42:20.5309576Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (2.8.2) 2022-11-23T01:42:20.5329875Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.12) 2022-11-23T01:42:20.5480517Z Requirement already satisfied: six>=1.5 in /home/ec2-user/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.16.0) 2022-11-23T01:42:20.7304736Z + python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test 2022-11-23T01:42:27.6466048Z [scribe] Scribe access token not provided, sending report via boto3... 2022-11-23T01:42:27.6466467Z 2022-11-23T01:42:27.6466930Z ----- Historic stats comparison result ------ 2022-11-23T01:42:27.6467261Z 2022-11-23T01:42:27.6467553Z job: linux-focal-py3.7-gcc7 2022-11-23T01:42:27.6468516Z commit: 1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T01:42:27.6469329Z 2022-11-23T01:42:27.6470056Z Commit graph (base is most recent master ancestor with at least one S3 report): 2022-11-23T01:42:27.6470388Z 2022-11-23T01:42:27.6470696Z : (master) 2022-11-23T01:42:27.6470956Z | 2022-11-23T01:42:27.6471662Z * 1cfd3858ac (HEAD) total time 1634.84s 2022-11-23T01:42:27.6490123Z * 26322544b8 (base) 3 reports, total time 1939.58s ± 1994.27s 2022-11-23T01:42:27.6490555Z * 7f4b4d2827 4 reports, total time 3007.47s ± 2710.74s 2022-11-23T01:42:27.6490862Z * b50699f247 6 reports, total time 3707.01s ± 3010.86s 2022-11-23T01:42:27.6491177Z * 8bf8e4d71e 6 reports, total time 3679.02s ± 2959.94s 2022-11-23T01:42:27.6491493Z * ce856cee7e 6 reports, total time 3732.77s ± 3067.45s 2022-11-23T01:42:27.6491789Z * 391b593ca2 6 reports, total time 3766.83s ± 3183.26s 2022-11-23T01:42:27.6492100Z * 5bba783d21 6 reports, total time 3688.41s ± 3016.52s 2022-11-23T01:42:27.6492465Z * ea920a1115 6 reports, total time 3713.47s ± 3039.67s 2022-11-23T01:42:27.6492766Z * 74e62a1fef 6 reports, total time 3657.38s ± 2942.16s 2022-11-23T01:42:27.6493073Z * 00b7d8ef23 6 reports, total time 3735.20s ± 2985.62s 2022-11-23T01:42:27.6493273Z | 2022-11-23T01:42:27.6493411Z : 2022-11-23T01:42:27.6493503Z 2022-11-23T01:42:27.6493626Z Removed (across 119 suites) 0 tests, totaling 0.00s 2022-11-23T01:42:27.6493890Z Modified (across 0 suites) 0 tests, totaling 0.00s 2022-11-23T01:42:27.6494149Z Added (across 103 suites) 976 tests, totaling +1634.84s 2022-11-23T01:42:27.7067730Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2022-11-23T01:42:27.7067975Z with: 2022-11-23T01:42:27.7068129Z env: 2022-11-23T01:42:27.7068291Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:42:27.7068571Z DOCKER_CONTAINER_ID: 6e200e7a94c0c354a3a0a6fc7e5a55f6d42c9daee3c19a19715f86f8daa96d25 2022-11-23T01:42:27.7068833Z ##[endgroup] 2022-11-23T01:42:27.7082117Z ##[group]Run set -eou pipefail 2022-11-23T01:42:27.7082353Z set -eou pipefail 2022-11-23T01:42:27.7082538Z  2022-11-23T01:42:27.7082761Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2022-11-23T01:42:27.7083016Z for _ in $(seq 1440); do 2022-11-23T01:42:27.7083239Z  # Break if no ssh session exists anymore 2022-11-23T01:42:27.7083461Z  if [ "$(who)" = "" ]; then 2022-11-23T01:42:27.7083635Z  break 2022-11-23T01:42:27.7083805Z  fi 2022-11-23T01:42:27.7083993Z  echo "." 2022-11-23T01:42:27.7084170Z  sleep 5 2022-11-23T01:42:27.7084340Z done 2022-11-23T01:42:27.7095332Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:42:27.7095553Z env: 2022-11-23T01:42:27.7095733Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:42:27.7096001Z DOCKER_CONTAINER_ID: 6e200e7a94c0c354a3a0a6fc7e5a55f6d42c9daee3c19a19715f86f8daa96d25 2022-11-23T01:42:27.7096351Z ##[endgroup] 2022-11-23T01:42:27.7120143Z Holding runner for 2 hours until all ssh sessions have logged out 2022-11-23T01:42:27.7210926Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2022-11-23T01:42:27.7211247Z # ignore expansion of "docker ps -q" since it could be empty 2022-11-23T01:42:27.7211500Z # shellcheck disable=SC2046 2022-11-23T01:42:27.7211728Z docker stop $(docker ps -q) || true 2022-11-23T01:42:27.7211941Z # Prune all of the docker images 2022-11-23T01:42:27.7212154Z docker system prune -af 2022-11-23T01:42:27.7222491Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:42:27.7222693Z env: 2022-11-23T01:42:27.7222868Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:42:27.7223151Z DOCKER_CONTAINER_ID: 6e200e7a94c0c354a3a0a6fc7e5a55f6d42c9daee3c19a19715f86f8daa96d25 2022-11-23T01:42:27.7223400Z ##[endgroup] 2022-11-23T01:42:28.1859450Z 6e200e7a94c0 2022-11-23T01:42:29.7117793Z Deleted Containers: 2022-11-23T01:42:29.7118106Z 6e200e7a94c0c354a3a0a6fc7e5a55f6d42c9daee3c19a19715f86f8daa96d25 2022-11-23T01:42:29.7118295Z 2022-11-23T01:42:36.4044665Z Deleted Images: 2022-11-23T01:42:36.4045601Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940 2022-11-23T01:42:36.4048540Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7@sha256:128a8060bc8d0d2006ae18b574265c3aa935b731c43947b53188b825fa718d6b 2022-11-23T01:42:36.4049339Z deleted: sha256:ac950c018cbcc41d6d7a73f33ac11fc7d5ae0cb1e9dbf1f471e7802e3c2787b3 2022-11-23T01:42:36.4049717Z deleted: sha256:b2992f8a5deb11d65e86729b600a9462d189b6a0e2d64ddf46a935b534edee73 2022-11-23T01:42:36.4050033Z deleted: sha256:377952e5d625f33006da79581ca002fe8708e712667cd20f83e3be62d7468554 2022-11-23T01:42:36.4050343Z deleted: sha256:9843138b6a57d457beab855e3460284d1866ba91a97f30314c279e5d91fe8d13 2022-11-23T01:42:36.4050654Z deleted: sha256:ef7ffe0d473b250807d16fbe59333f43b284836e2f113fc021b51c63b2e011e7 2022-11-23T01:42:36.4050983Z deleted: sha256:36e6aaa83981278ab4c62090dd61d8e6b9646d6d29d3e8cf7fe3fa5d9513cf0a 2022-11-23T01:42:36.4051304Z deleted: sha256:547328983788be2665bb6e828066d6feeee1e5b15bf2ddc6bf2f259552437c2a 2022-11-23T01:42:36.4051617Z deleted: sha256:e2e21028b051073a8bc6980590edf442bdc3e1306e0028b49dfd55cf2c91b9ee 2022-11-23T01:42:36.4051963Z deleted: sha256:43b659b86bc116dbce677f866a791ec472a054d893eceadb868a83fc10ddf981 2022-11-23T01:42:36.4052442Z deleted: sha256:aee61630fbc682636784477102a9b6b79fe2d838f21e59d50668d685fdb299f8 2022-11-23T01:42:36.4052764Z deleted: sha256:dc980582b7d0ab6953bd378bfa7248ef8e950d4463bf46788b8c9fdd38de09c6 2022-11-23T01:42:36.4053096Z deleted: sha256:bc41e9d5653f1c334e1531d0b8f6a3c2c71eb1b5faa6ccf4f1982e341f3f51b7 2022-11-23T01:42:36.4053423Z deleted: sha256:cc9cae77052e164c6a73da731c9b62ff3d69b8ab2ec5e497ad84e27f512755a3 2022-11-23T01:42:36.4053751Z deleted: sha256:90bb77a23d716774ba1e5e2088208bd8737f96f2bbfb9a08e784f7251c6f76d6 2022-11-23T01:42:36.4054066Z deleted: sha256:8964a3bf610c6b0f62ab78a367344a7dd597b5c72219923f1e09001e2e3b79f3 2022-11-23T01:42:36.4054357Z deleted: sha256:b75210d5af1517b607b80542cb9a815c9703720427a8b559e399cb7a2a7ba8a7 2022-11-23T01:42:36.4054672Z deleted: sha256:14f0bcbce4ae20f68c2e993ad2746d59f4f456c783d5862b894e7245ff2940f4 2022-11-23T01:42:36.4055014Z deleted: sha256:9b6bcbca6fcb1af72a85da5f16b8295e3c678611151b4d16f9a1575fe3db8492 2022-11-23T01:42:36.4055349Z deleted: sha256:f199e2cb9461ed2044209904fafeb4de59b7c08fea775715b633eaaa1987d76a 2022-11-23T01:42:36.4055648Z deleted: sha256:70b4e3728651e26c75a4845e785a5e2e91e3a7d3203833638a093f2e1d8b221a 2022-11-23T01:42:36.4055950Z deleted: sha256:1d5959fe962bc85cd740541c582b145ce1374b02b323beff68a0426d63c86a49 2022-11-23T01:42:36.4056256Z deleted: sha256:87062382eaa8d09a8a232a7d4586e0241eb319eb75654d7d652183e890363f0a 2022-11-23T01:42:36.4056535Z deleted: sha256:31666c108535eb4050261a140e3f31726c89b938120e1e39ff1d81c8cab6a349 2022-11-23T01:42:36.4056927Z deleted: sha256:98a161950ac2230e581a5e15e4905b1855e6ee1fe94558d8473e2e65e375475f 2022-11-23T01:42:36.4057247Z deleted: sha256:c4276d6adb8f7a22a7dbacdc0f6552113ccc525a19608c9f201399a95862928c 2022-11-23T01:42:36.4057576Z deleted: sha256:61a0274c3c6c91a0cd553a026ed8b8f1a924c35295bb41c32927f7e6e4660d39 2022-11-23T01:42:36.4057891Z deleted: sha256:57f2baf11e15f4b61fd2de15d0f7c10150e83aca1de166f644c472b2780428e7 2022-11-23T01:42:36.4058214Z deleted: sha256:2a5723d4a8c0331709b4ff22a4d78f12e5e1396a281832c1c7f1ded6257c06b6 2022-11-23T01:42:36.4058531Z deleted: sha256:b235c0c7e72494541107ca95d5d5445eb1e94fbd4ce40b73d7d98010105de22a 2022-11-23T01:42:36.4058828Z deleted: sha256:74e83991baa883fd71a6668d703f0419506e79c602d19f6888efc9ee60eda5a0 2022-11-23T01:42:36.4059153Z deleted: sha256:149935fef489f40eadd6a9475abd8bd67700c1bc9fdff273fdfcc31ef4baa465 2022-11-23T01:42:36.4059488Z deleted: sha256:635c2c6690b7bbedfda868b6fb88cc966a2c005b9540839fa78c30bf0e8d0e08 2022-11-23T01:42:36.4059825Z deleted: sha256:c962aa8089ba331791c2b2b83740f4db5b1aac8ef8e891cdc595c3fdb5310dbc 2022-11-23T01:42:36.4060133Z deleted: sha256:b51083cce1d6de548b12fd7b666205f728d375ce6124109e79212b44fb198d87 2022-11-23T01:42:36.4060454Z deleted: sha256:e3c257e1d8787783e8f79a25fcd497cce01bd7bedcf407b8eae290d81428f1ca 2022-11-23T01:42:36.4060781Z deleted: sha256:68698630dd71c4db0eacdac941d296ab20d7c4370c8122793a2d452f0c6f476d 2022-11-23T01:42:36.4061083Z deleted: sha256:aca108602e826c17f32620b9ade1562004117a689df4bb1b4ae46b638a36c2a1 2022-11-23T01:42:36.4061389Z deleted: sha256:93b007516b03c7ea1975114590b8e43ae2e9309180acc0ee20b586dc41ff0e4c 2022-11-23T01:42:36.4061703Z deleted: sha256:4b0b6cc64e7afa0bebb597b5bb5755852430b284745950b748fdf945ee7c7343 2022-11-23T01:42:36.4062007Z deleted: sha256:510466f40335397d70041861d895d661aba31b0f89fe9ed823b502709e6a048c 2022-11-23T01:42:36.4062306Z deleted: sha256:a940b4fb25d752aed6e2effcfdd74338042ad2d2f1536438dc53a065b609d3b1 2022-11-23T01:42:36.4062632Z deleted: sha256:effe44ab85505068890636f69282d6246ca2c13d558b18d5ff52d44acedf52e4 2022-11-23T01:42:36.4062962Z deleted: sha256:2b26ee6f3f7e8b943ac9ae9d70b6fff3e4ebd4b5c42d530a5939aab7a5a8e968 2022-11-23T01:42:36.4063287Z deleted: sha256:b86b5801635d9fb03bccba09570b1c9c770d5eb6e6d1a1efde30df10444cbcda 2022-11-23T01:42:36.4063612Z deleted: sha256:491df340aee36942648573aeef75d3f1ece9f65d74f7d6a025b41ac68b17774b 2022-11-23T01:42:36.4063927Z deleted: sha256:e10409ac2ee50fb36987bc7932b127299cffc29888a5917ba29c36024f6fd3b6 2022-11-23T01:42:36.4064261Z deleted: sha256:eecb37d5ef051b777dce74f18fff350cc8c9c687f178d4321a1f4ba3a66f8e17 2022-11-23T01:42:36.4064632Z deleted: sha256:c033ce386c9769757faf2693533fea2abcd3e7c50f538aed3d0202907220131a 2022-11-23T01:42:36.4064955Z deleted: sha256:01cef2346a033872b4dc7c3938eb6be98e8c9b54ec05d98b84081c122f21d0e5 2022-11-23T01:42:36.4065280Z deleted: sha256:743ac6a4351c56c1d41bf297ddb1978e237e87ce4f78ef5eae80927c64b73c6e 2022-11-23T01:42:36.4065625Z deleted: sha256:6ae19417bc8a33832e82e9051aeca485e2839a197ed86584713056b959d53f6e 2022-11-23T01:42:36.4065964Z deleted: sha256:bd69c9e420df78782dec5b44ac56d41ee0fcb2deb00d6b2b439e5a33b79eb5d9 2022-11-23T01:42:36.4066311Z deleted: sha256:26986e9c9bf97e7a71da10396a6069f1d61d84cffa373ef9295ae547857faf74 2022-11-23T01:42:36.4066728Z deleted: sha256:a2fade5710ceea279a50353534afd459a814b185fe0f4a2baa00ed296a7e73a9 2022-11-23T01:42:36.4067168Z deleted: sha256:07108459e4c8e804d5f79b0a75a3902a681d1d77f1651dac39c5944cc9070846 2022-11-23T01:42:36.4067569Z deleted: sha256:afde38d23fe01a4fe20b0c36952db88f94e3ae5f90196a1f0da7f668ced05a1a 2022-11-23T01:42:36.4067913Z deleted: sha256:329b465bb32a9dc01247f25c90efd8ce5a2d1ca603c32f17c6becbf911de8ea9 2022-11-23T01:42:36.4068229Z deleted: sha256:2bf3e548b369a3c5003bf543698d18f933ece97bc5bcfd8bb7454dd4f3c02ec1 2022-11-23T01:42:36.4068553Z deleted: sha256:f4462d5b2da2985f37409c9b257afd2b9fb82356ce4e43e804ee34214242e34a 2022-11-23T01:42:36.4068731Z 2022-11-23T01:42:36.4068835Z Total reclaimed space: 9.775GB 2022-11-23T01:42:36.4148212Z Post job cleanup. 2022-11-23T01:42:36.4191623Z Post job cleanup. 2022-11-23T01:42:36.5242503Z [command]/usr/bin/git version 2022-11-23T01:42:36.5285944Z git version 2.37.1 2022-11-23T01:42:36.5324782Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/67951dc6-f36b-4348-8842-ff7ec9995a75' before making global git config changes 2022-11-23T01:42:36.5325404Z Adding repository directory to the temporary git global config as a safe directory 2022-11-23T01:42:36.5329052Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2022-11-23T01:42:36.5374703Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2022-11-23T01:42:36.5404657Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || : 2022-11-23T01:42:36.5824970Z Entering 'android/libs/fbjni' 2022-11-23T01:42:36.5868950Z Entering 'third_party/FP16' 2022-11-23T01:42:36.5913738Z Entering 'third_party/FXdiv' 2022-11-23T01:42:36.5951211Z Entering 'third_party/NNPACK' 2022-11-23T01:42:36.5998572Z Entering 'third_party/QNNPACK' 2022-11-23T01:42:36.6045260Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T01:42:36.6080743Z Entering 'third_party/XNNPACK' 2022-11-23T01:42:36.6185968Z Entering 'third_party/benchmark' 2022-11-23T01:42:36.6231604Z Entering 'third_party/cpuinfo' 2022-11-23T01:42:36.6274980Z Entering 'third_party/cub' 2022-11-23T01:42:36.6321634Z Entering 'third_party/cudnn_frontend' 2022-11-23T01:42:36.6361006Z Entering 'third_party/cutlass' 2022-11-23T01:42:36.6401581Z Entering 'third_party/eigen' 2022-11-23T01:42:36.6462648Z Entering 'third_party/fbgemm' 2022-11-23T01:42:36.6522786Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T01:42:36.6577445Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T01:42:36.6630880Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T01:42:36.6683560Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T01:42:36.6722568Z Entering 'third_party/flatbuffers' 2022-11-23T01:42:36.6758463Z Entering 'third_party/fmt' 2022-11-23T01:42:36.6793188Z Entering 'third_party/foxi' 2022-11-23T01:42:36.6827693Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T01:42:36.6861498Z Entering 'third_party/gloo' 2022-11-23T01:42:36.6896115Z Entering 'third_party/googletest' 2022-11-23T01:42:36.6931960Z Entering 'third_party/ideep' 2022-11-23T01:42:36.6966086Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T01:42:36.7001407Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T01:42:36.7041649Z Entering 'third_party/ios-cmake' 2022-11-23T01:42:36.7075698Z Entering 'third_party/ittapi' 2022-11-23T01:42:36.7110324Z Entering 'third_party/kineto' 2022-11-23T01:42:36.7144315Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T01:42:36.7178169Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T01:42:36.7213975Z Entering 'third_party/nccl/nccl' 2022-11-23T01:42:36.7249395Z Entering 'third_party/neon2sse' 2022-11-23T01:42:36.7283723Z Entering 'third_party/nlohmann' 2022-11-23T01:42:36.7318861Z Entering 'third_party/onnx' 2022-11-23T01:42:36.7364861Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T01:42:36.7399453Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T01:42:36.7435858Z Entering 'third_party/onnx-tensorrt' 2022-11-23T01:42:36.7469850Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T01:42:36.7508637Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T01:42:36.7543721Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T01:42:36.7580436Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T01:42:36.7619302Z Entering 'third_party/pocketfft' 2022-11-23T01:42:36.7653865Z Entering 'third_party/protobuf' 2022-11-23T01:42:36.7692130Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T01:42:36.7725277Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T01:42:36.7760898Z Entering 'third_party/psimd' 2022-11-23T01:42:36.7796157Z Entering 'third_party/pthreadpool' 2022-11-23T01:42:36.7830568Z Entering 'third_party/pybind11' 2022-11-23T01:42:36.7865040Z Entering 'third_party/python-enum' 2022-11-23T01:42:36.7898969Z Entering 'third_party/python-peachpy' 2022-11-23T01:42:36.7934212Z Entering 'third_party/python-six' 2022-11-23T01:42:36.7969631Z Entering 'third_party/sleef' 2022-11-23T01:42:36.8004988Z Entering 'third_party/tbb' 2022-11-23T01:42:36.8042010Z Entering 'third_party/tensorpipe' 2022-11-23T01:42:36.8076757Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T01:42:36.8110282Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T01:42:36.8143608Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T01:42:36.8176375Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T01:42:36.8210063Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T01:42:36.8246698Z Entering 'third_party/zstd' 2022-11-23T01:42:36.8295509Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2022-11-23T01:42:36.8320876Z http.https://github.com/.extraheader 2022-11-23T01:42:36.8328390Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2022-11-23T01:42:36.8359451Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || : 2022-11-23T01:42:36.8615471Z Entering 'android/libs/fbjni' 2022-11-23T01:42:36.8634844Z http.https://github.com/.extraheader 2022-11-23T01:42:36.8662278Z Entering 'third_party/FP16' 2022-11-23T01:42:36.8683338Z http.https://github.com/.extraheader 2022-11-23T01:42:36.8710419Z Entering 'third_party/FXdiv' 2022-11-23T01:42:36.8730105Z http.https://github.com/.extraheader 2022-11-23T01:42:36.8757613Z Entering 'third_party/NNPACK' 2022-11-23T01:42:36.8779891Z http.https://github.com/.extraheader 2022-11-23T01:42:36.8807854Z Entering 'third_party/QNNPACK' 2022-11-23T01:42:36.8828882Z http.https://github.com/.extraheader 2022-11-23T01:42:36.8866089Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T01:42:36.8886531Z http.https://github.com/.extraheader 2022-11-23T01:42:36.8913794Z Entering 'third_party/XNNPACK' 2022-11-23T01:42:36.8934923Z http.https://github.com/.extraheader 2022-11-23T01:42:36.8972846Z Entering 'third_party/benchmark' 2022-11-23T01:42:36.8993345Z http.https://github.com/.extraheader 2022-11-23T01:42:36.9020548Z Entering 'third_party/cpuinfo' 2022-11-23T01:42:36.9040762Z http.https://github.com/.extraheader 2022-11-23T01:42:36.9068533Z Entering 'third_party/cub' 2022-11-23T01:42:36.9089538Z http.https://github.com/.extraheader 2022-11-23T01:42:36.9115861Z Entering 'third_party/cudnn_frontend' 2022-11-23T01:42:36.9135835Z http.https://github.com/.extraheader 2022-11-23T01:42:36.9168377Z Entering 'third_party/cutlass' 2022-11-23T01:42:36.9188454Z http.https://github.com/.extraheader 2022-11-23T01:42:36.9223194Z Entering 'third_party/eigen' 2022-11-23T01:42:36.9243777Z http.https://github.com/.extraheader 2022-11-23T01:42:36.9273467Z Entering 'third_party/fbgemm' 2022-11-23T01:42:36.9294227Z http.https://github.com/.extraheader 2022-11-23T01:42:36.9321823Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T01:42:36.9341196Z http.https://github.com/.extraheader 2022-11-23T01:42:36.9369192Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T01:42:36.9388564Z http.https://github.com/.extraheader 2022-11-23T01:42:36.9415588Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T01:42:36.9435897Z http.https://github.com/.extraheader 2022-11-23T01:42:36.9463659Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T01:42:36.9482960Z http.https://github.com/.extraheader 2022-11-23T01:42:36.9511294Z Entering 'third_party/flatbuffers' 2022-11-23T01:42:36.9531318Z http.https://github.com/.extraheader 2022-11-23T01:42:36.9560553Z Entering 'third_party/fmt' 2022-11-23T01:42:36.9580652Z http.https://github.com/.extraheader 2022-11-23T01:42:36.9608351Z Entering 'third_party/foxi' 2022-11-23T01:42:36.9628534Z http.https://github.com/.extraheader 2022-11-23T01:42:36.9655901Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T01:42:36.9676743Z http.https://github.com/.extraheader 2022-11-23T01:42:36.9703722Z Entering 'third_party/gloo' 2022-11-23T01:42:36.9724917Z http.https://github.com/.extraheader 2022-11-23T01:42:36.9751325Z Entering 'third_party/googletest' 2022-11-23T01:42:36.9771470Z http.https://github.com/.extraheader 2022-11-23T01:42:36.9797888Z Entering 'third_party/ideep' 2022-11-23T01:42:36.9817741Z http.https://github.com/.extraheader 2022-11-23T01:42:36.9852309Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T01:42:36.9871638Z http.https://github.com/.extraheader 2022-11-23T01:42:36.9912076Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T01:42:36.9931807Z http.https://github.com/.extraheader 2022-11-23T01:42:36.9973803Z Entering 'third_party/ios-cmake' 2022-11-23T01:42:36.9993714Z http.https://github.com/.extraheader 2022-11-23T01:42:37.0020177Z Entering 'third_party/ittapi' 2022-11-23T01:42:37.0041219Z http.https://github.com/.extraheader 2022-11-23T01:42:37.0067979Z Entering 'third_party/kineto' 2022-11-23T01:42:37.0090414Z http.https://github.com/.extraheader 2022-11-23T01:42:37.0117386Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T01:42:37.0137353Z http.https://github.com/.extraheader 2022-11-23T01:42:37.0165652Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T01:42:37.0185742Z http.https://github.com/.extraheader 2022-11-23T01:42:37.0214340Z Entering 'third_party/nccl/nccl' 2022-11-23T01:42:37.0234039Z http.https://github.com/.extraheader 2022-11-23T01:42:37.0260863Z Entering 'third_party/neon2sse' 2022-11-23T01:42:37.0282023Z http.https://github.com/.extraheader 2022-11-23T01:42:37.0308189Z Entering 'third_party/nlohmann' 2022-11-23T01:42:37.0328520Z http.https://github.com/.extraheader 2022-11-23T01:42:37.0356839Z Entering 'third_party/onnx' 2022-11-23T01:42:37.0376758Z http.https://github.com/.extraheader 2022-11-23T01:42:37.0415887Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T01:42:37.0435652Z http.https://github.com/.extraheader 2022-11-23T01:42:37.0470747Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T01:42:37.0491170Z http.https://github.com/.extraheader 2022-11-23T01:42:37.0532264Z Entering 'third_party/onnx-tensorrt' 2022-11-23T01:42:37.0551849Z http.https://github.com/.extraheader 2022-11-23T01:42:37.0578827Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T01:42:37.0599184Z http.https://github.com/.extraheader 2022-11-23T01:42:37.0643182Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T01:42:37.0662398Z http.https://github.com/.extraheader 2022-11-23T01:42:37.0690443Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T01:42:37.0709389Z http.https://github.com/.extraheader 2022-11-23T01:42:37.0736460Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T01:42:37.0755509Z http.https://github.com/.extraheader 2022-11-23T01:42:37.0797102Z Entering 'third_party/pocketfft' 2022-11-23T01:42:37.0817613Z http.https://github.com/.extraheader 2022-11-23T01:42:37.0844463Z Entering 'third_party/protobuf' 2022-11-23T01:42:37.0863734Z http.https://github.com/.extraheader 2022-11-23T01:42:37.0893918Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T01:42:37.0913272Z http.https://github.com/.extraheader 2022-11-23T01:42:37.0940095Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T01:42:37.0959745Z http.https://github.com/.extraheader 2022-11-23T01:42:37.0988809Z Entering 'third_party/psimd' 2022-11-23T01:42:37.1009228Z http.https://github.com/.extraheader 2022-11-23T01:42:37.1035352Z Entering 'third_party/pthreadpool' 2022-11-23T01:42:37.1054973Z http.https://github.com/.extraheader 2022-11-23T01:42:37.1082983Z Entering 'third_party/pybind11' 2022-11-23T01:42:37.1102203Z http.https://github.com/.extraheader 2022-11-23T01:42:37.1142397Z Entering 'third_party/python-enum' 2022-11-23T01:42:37.1164914Z http.https://github.com/.extraheader 2022-11-23T01:42:37.1191559Z Entering 'third_party/python-peachpy' 2022-11-23T01:42:37.1211045Z http.https://github.com/.extraheader 2022-11-23T01:42:37.1237563Z Entering 'third_party/python-six' 2022-11-23T01:42:37.1258099Z http.https://github.com/.extraheader 2022-11-23T01:42:37.1285506Z Entering 'third_party/sleef' 2022-11-23T01:42:37.1305341Z http.https://github.com/.extraheader 2022-11-23T01:42:37.1332768Z Entering 'third_party/tbb' 2022-11-23T01:42:37.1352747Z http.https://github.com/.extraheader 2022-11-23T01:42:37.1382043Z Entering 'third_party/tensorpipe' 2022-11-23T01:42:37.1402301Z http.https://github.com/.extraheader 2022-11-23T01:42:37.1428992Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T01:42:37.1449147Z http.https://github.com/.extraheader 2022-11-23T01:42:37.1476145Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T01:42:37.1495532Z http.https://github.com/.extraheader 2022-11-23T01:42:37.1522054Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T01:42:37.1541823Z http.https://github.com/.extraheader 2022-11-23T01:42:37.1570628Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T01:42:37.1589646Z http.https://github.com/.extraheader 2022-11-23T01:42:37.1615951Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T01:42:37.1635547Z http.https://github.com/.extraheader 2022-11-23T01:42:37.1665437Z Entering 'third_party/zstd' 2022-11-23T01:42:37.1685902Z http.https://github.com/.extraheader 2022-11-23T01:42:37.1914347Z Cleaning up orphan processes